This is gnuastro.info, produced by makeinfo version 7.0.3 from
gnuastro.texi.

This book documents version 0.21 of the GNU Astronomy Utilities
(Gnuastro).  Gnuastro provides various programs and libraries for
astronomical data manipulation and analysis.

   Copyright © 2015-2023 Free Software Foundation, Inc.

     Permission is granted to copy, distribute and/or modify this
     document under the terms of the GNU Free Documentation License,
     Version 1.3 or any later version published by the Free Software
     Foundation; with no Invariant Sections, no Front-Cover Texts, and
     no Back-Cover Texts.  A copy of the license is included in the
     section entitled “GNU Free Documentation License”.
INFO-DIR-SECTION Astronomy
START-INFO-DIR-ENTRY
* Gnuastro: (gnuastro).       GNU Astronomy Utilities.
* libgnuastro: (gnuastro)Gnuastro library. Full Gnuastro library doc.

* help-gnuastro: (gnuastro)help-gnuastro mailing list. Getting help.

* bug-gnuastro: (gnuastro)Report a bug. How to report bugs

* Arithmetic: (gnuastro)Arithmetic. Arithmetic operations on pixels.
* astarithmetic: (gnuastro)Invoking astarithmetic. Options to Arithmetic.

* BuildProgram: (gnuastro)BuildProgram. Compile and run programs using Gnuastro’s library.
* astbuildprog: (gnuastro)Invoking astbuildprog. Options to BuildProgram.

* ConvertType: (gnuastro)ConvertType. Convert different file types.
* astconvertt: (gnuastro)Invoking astconvertt. Options to ConvertType.

* Convolve: (gnuastro)Convolve. Convolve an input file with kernel.
* astconvolve: (gnuastro)Invoking astconvolve. Options to Convolve.

* CosmicCalculator: (gnuastro)CosmicCalculator. For cosmological params.
* astcosmiccal: (gnuastro)Invoking astcosmiccal. Options to CosmicCalculator.

* Crop: (gnuastro)Crop. Crop region(s) from image(s).
* astcrop: (gnuastro)Invoking astcrop. Options to Crop.

* Fits: (gnuastro)Fits. View and manipulate FITS extensions and keywords.
* astfits: (gnuastro)Invoking astfits. Options to Fits.

* MakeCatalog: (gnuastro)MakeCatalog. Make a catalog from labeled image.
* astmkcatalog: (gnuastro)Invoking astmkcatalog. Options to MakeCatalog.

* MakeProfiles: (gnuastro)MakeProfiles. Make mock profiles.
* astmkprof: (gnuastro)Invoking astmkprof. Options to MakeProfiles.

* Match: (gnuastro)Match. Match two separate catalogs.
* astmatch: (gnuastro)Invoking astmatch. Options to Match.

* NoiseChisel: (gnuastro)NoiseChisel. Detect signal in noise.
* astnoisechisel: (gnuastro)Invoking astnoisechisel. Options to NoiseChisel.

* Segment: (gnuastro)Segment. Segment detections based on signal structure.
* astsegment: (gnuastro)Invoking astsegment. Options to Segment.

* Query: (gnuastro)Query. Access remote databases for downloading data.
* astquery: (gnuastro)Invoking astquery. Options to Query.

* Statistics: (gnuastro)Statistics. Get image Statistics.
* aststatistics: (gnuastro)Invoking aststatistics. Options to Statistics.

* Table: (gnuastro)Table. Read and write FITS binary or ASCII tables.
* asttable: (gnuastro)Invoking asttable. Options to Table.

* Warp: (gnuastro)Warp. Warp a dataset to a new grid.
* astwarp: (gnuastro)Invoking astwarp. Options to Warp.

* astscript: (gnuastro)Installed scripts. Gnuastro’s installed scripts.
* astscript-ds9-region: (gnuastro)Invoking astscript-ds9-region. Options to this script
* astscript-fits-view: (gnuastro)Invoking astscript-fits-view. Options to this script
* astscript-pointing-simulate: (gnuastro)Invoking astscript-pointing-simulate. Options to this script
* astscript-psf-scale-factor: (gnuastro)Invoking astscript-psf-scale-factor. Options to this script
* astscript-psf-select-stars: (gnuastro)Invoking astscript-psf-select-stars. Options to this script
* astscript-psf-stamp: (gnuastro)Invoking astscript-psf-stamp. Options to this script
* astscript-psf-subtract: (gnuastro)Invoking astscript-psf-subtract. Options to this script
* astscript-psf-unite: (gnuastro)Invoking astscript-psf-unite. Options to this script
* astscript-radial-profile: (gnuastro)Invoking astscript-radial-profile. Options to this script
* astscript-sort-by-night: (gnuastro)Invoking astscript-sort-by-night. Options to this script
* astscript-zeropoint: (gnuastro)Invoking astscript-zeropoint. Options to this script
END-INFO-DIR-ENTRY


File: gnuastro.info,  Node: Moire pattern in stacking and its correction,  Prev: Pointing pattern design,  Up: Tutorials

2.9 Moiré pattern in stacking and its correction
================================================

After warping some images with the default mode of Warp (see *note Align
pixels with WCS considering distortions::) you may notice that the
background noise is no longer flat.  Some regions will be smoother and
some will be sharper; depending on the orientation and distortion of the
input/output pixel grids.  This is due to the Moiré pattern
(https://en.wikipedia.org/wiki/Moir%C3%A9_pattern), which is especially
noticeable/significant when two slightly different grids are
super-imposed.

   With the commands below, we’ll download a single exposure image from
the J-PLUS survey (https://www.j-plus.es) and run Warp (on a $8\times8$
arcmin$^2$ region to speed it up the demos here).  Finally, we’ll open
the image to visually see the artificial Moiré pattern on the warped
image.

     ## Download the image (73.7 MB containing an 9216x9232 pixel image)
     $ jplusdr2=http://archive.cefca.es/catalogues/vo/siap/jplus-dr2/reduced
     $ wget $jplusdr2/get_fits?id=771463 -Ojplus-exp1.fits.fz

     ## Align a small part of it with the sky coordinates.
     $ astwarp jplus-exp1.fits.fz --center=107.62920,39.72472 \
               --width=8/60 -ojplus-e1.fits

     ## Open the aligned region with DS9
     $ astscript-fits-view jplus-e1.fits

   In the opened DS9 window, you can see the Moiré pattern as wave-like
patterns in the noise: some parts of the noise are more smooth and some
parts are more sharp.  Right in the center of the image is a blob of
sharp noise.  Warp has the ‘--checkmaxfrac’ option for direct inspection
of the Moiré pattern (described with the other options in *note Align
pixels with WCS considering distortions::).  When run with this option,
an extra HDU (called ‘MAX-FRAC’) will be added to the output.  The image
in this HDU has the same size as the output.  However, each output pixel
will contain the largest (maximum) fraction of area that it covered over
the input pixel grid.  So if an output pixel has a value of 0.9, this
shows that it covered $90\%$ of an input pixel.  Let’s run Warp with
‘--checkmaxfrac’ and see the output (after DS9 opens, in the “Cube”
window, flip between the first and second HDUs):

     $ astwarp jplus-exp1.fits.fz --center=107.62920,39.72472 \
               --width=8/60 -ojplus-e1.fits --checkmaxfrac

     $ astscript-fits-view jplus-e1.fits

   By comparing the first and second HDUs/extensions, you will clearly
see that the regions with a sharp noise pattern fall exactly on parts of
the ‘MAX-FRAC’ extension with values larger than 0.5.  In other words,
output pixels where one input pixel contributed more than half of the
its value.  As this fraction increases, the sharpness also increases
because a single input pixel’s value dominates the value of the output
pixel.  On the other hand, when this value is small, we see that many
input pixels contribute to that output pixel.  Since many input pixels
contribute to an output pixel, it acts like a convolution, hence that
output pixel becomes smoother (see *note Spatial domain convolution::).
Let’s have a look at the distribution of the ‘MAX-FRAC’ pixel values:

     $ aststatistics jplus-e1.fits -hMAX-FRAC
     Statistics (GNU Astronomy Utilities) 0.21
     -------
     Input: jplus-e1.fits (hdu: MAX-FRAC)
     -------
       Number of elements:                      744769
       Minimum:                                 0.250213461
       Maximum:                                 0.9987495374
       Mode:                                    0.5034223567
       Mode quantile:                           0.3773819498
       Median:                                  0.5520805544
       Mean:                                    0.5693956458
       Standard deviation:                      0.1554693738
     -------
     Histogram:
      |                      ***
      |                   **********
      |                 *****************
      |              ************************
      |           *******************************
      |         **************************************
      |       *********************************************
      |     ****************************************************
      |   ***********************************************************
      | ******************************************************************
      |**********************************************************************
      |----------------------------------------------------------------------

   The smallest value is 0.25 (=1/4), showing that 4 input pixels
contributed to the output pixels value.  While the maximum is almost
1.0, showing that a single input pixel defined the output pixel value.
You can also see that the most probable value (the mode) is 0.5, and
that the distribution is positively skewed.

   This is a well-known problem in astronomical imaging and professional
photography.  If you only have a single image (that is already taken!),
you can undersample the input: set the angular size of the output pixels
to be larger than the input.  This will decrease the resolution of your
image, but will ensure that pixel-mixing will always happen.  In the
example below we are setting the output pixel scale (which is known as
‘CDELT’ in the FITS standard) to $1/0.5=2$ of the input’s.  In other
words each output pixel edge will cover double the input pixel’s edge on
the sky, and the output’s number of pixels in each dimension will be
half of the previous output.

     $ cdelt=$(astfits jplus-exp1.fits.fz --pixelscale -q \
                       | awk '{print $1}')
     $ astwarp jplus-exp1.fits.fz --center=107.62920,39.72472 \
               --width=8/60 -ojplus-e1.fits --cdelt=$cdelt/0.5 \
               --checkmaxfrac

   In the first extension, you can hardly see any Moiré pattern in the
noise.  When you go to the next (‘MAX-FRAC’) extension, you will see
that almost all the pixels have a value of 1.  Of course, decreasing the
resolution by half is a little too drastic.  Depending on your image,
you may be able to reach a sufficiently good result without such a
drastic degrading of the input image.  For example, if you want an
output pixel scale that is just 1.5 times larger than the input, you can
divide the original coordinate-delta (or “cdelt”) by $1/1.5=0.6666$ and
try again.  In the ‘MAX-FRAC’ extension, you will see that the range of
pixel values is now between 0.56 to 1.0 (recall that originally, this
was between 0.25 and 1.0).  This shows that the pixels are more
similarly mixed and in fact, when you look at the actual warped image,
you can hardly distinguish any Moiré pattern in the noise.

   However, deep astronomical data are usually built by several
exposures (images), not a single one.  Each image is also taken by
(slightly) shifting the telescope compared to the previous exposure.
This shift is known as “dithering” or a “pointing pattern”, see *note
Pointing pattern design::.  We do this for many reasons (for example
tracking errors in the telescope, high background values, removing the
effect of bad pixels or those affected by cosmic rays, robust flat
pattern measurement, etc.(1)).  One of those “etc.” reasons is to
correct the Moiré pattern in the final coadded deep image.

   The Moiré pattern is fixed to the grid of the image, slightly
shifting the telescope will result in the pattern appearing in different
parts of the sky.  Therefore when we later stack, or coadd, the separate
exposures into a deep image, the Moiré pattern will be decreased there.
However, dithering has possible drawbacks based on the scientific goal.
For example when observing time-variable phenomena where cutting the
exposures to several shorter ones is not feasible.  If this is not the
case for you (for example in galaxy evolution), continue with the rest
of this section.

   Because we have multiple exposures that are slightly (sub-pixel)
shifted, we can also increase the spatial resolution of the output.  For
example, let’s set the output coordinate-delta (or pixel scale) to be
1/2 of the input.  In other words, the number of pixels in each
dimension of the output is double the first Warp command of this
section:

     $ astwarp jplus-exp1.fits.fz --center=107.62920,39.72472 \
               --width=8/60 -ojplus-e1.fits --cdelt=$cdelt/2 \
               --checkmaxfrac

     $ aststatistics jplus-e1.fits -hMAX-FRAC --minimum --maximum
     0.06263604388 0.2506802701

     $ astscript-fits-view jplus-e1.fits

   From the last command, you see that like the previous change in
‘--cdelt’, the range of ‘MAX-FRAC’ has decreased.  However, when you
look at the warped image and the ‘MAX-FRAC’ image with the last command,
you still visually see the Moiré pattern in the noise (although it has
significantly decreased compared to the original resolution).  It is
still present because 2 is an exact multiple of 1.  Let’s try increasing
the resolution (oversampling) by a factor of 1.25 (which isn’t an exact
multiple of 1):

     $ astwarp jplus-exp1.fits.fz --center=107.62920,39.72472 \
               --width=8/60 -ojplus-e1.fits --cdelt=$cdelt/1.25 \
               --checkmaxfrac
     $ astscript-fits-view jplus-e1.fits

   You don’t see any Moiré pattern in the noise any more, but when you
look at the ‘MAX-FRAC’ extension, you see it is very different from the
ones you had seen before.  In the previous ‘MAX-FRAC’ image, you could
see large blobs of similar values.  But here, you see that the variation
is almost on a pixel scale, and the difference between one pixel to the
next is not significant.  This is why you don’t see any Moiré pattern in
the warped image.

   In J-PLUS, each part of the sky was observed with a three-point
pointing pattern (very small shifts in each pointing).  Let’s download
the other two exposures and warp the same region of the sky to the same
pixel grid (using the ‘--gridfile’ feature).  Then, let’s open all three
cropped images in one DS9 instance:

     $ wget $jplusdr2/get_fits?id=771465 -Ojplus-exp2.fits.fz
     $ wget $jplusdr2/get_fits?id=771467 -Ojplus-exp3.fits.fz

     $ astwarp jplus-exp2.fits.fz --gridfile jplus-e1.fits \
               -o jplus-e2.fits --checkmaxfrac
     $ astwarp jplus-exp3.fits.fz --gridfile jplus-e1.fits \
               -o jplus-e3.fits --checkmaxfrac

     $ astscript-fits-view jplus-e*.fits

In the three warped images, you don’t see any Moiré pattern, so far so
good...  now, take the following steps:
  1. Click on the “Frame” button (in the top row of buttons just on top
     of the image), and select the “Single” button in the bottom row.
  2. Open the “Zoom” menu, and select “Zoom 16”.
  3. In the bottom row of buttons right on top of the image, press the
     “next” button to flip through each exposure’s ‘MAX-FRAC’ extension.
  4. Focus your eyes on the pixels with the largest value (white colored
     pixels), while pressing the “next” button to flip between the
     exposures.  You will see that in each exposure they cover different
     pixels.

   The exercise above shows that the effect varying smoothing level
(that had already shrank to a per-pixel level) will be further decreased
after we stack the images.  So let’s stack these three images with the
commands below.  First, we need to remove the sky-level from each image
using *note NoiseChisel::, then we’ll stack the ‘INPUT-NO-SKY’
extensions using sigma-clipping (to reject outliers by *note Sigma
clipping::, using the *note Stacking operators::).

     $ astnoisechisel jplus-e1.fits -ojplus-nc1.fits
     $ astnoisechisel jplus-e2.fits -ojplus-nc2.fits
     $ astnoisechisel jplus-e3.fits -ojplus-nc3.fits

     $ astarithmetic jplus-nc*.fits 3 5 0.2 sigclip-mean \
                     -gINPUT-NO-SKY -ojplus-stack.fits

     $ astscript-fits-view jplus-nc*.fits jplus-stack.fits

After opening the individual exposures and the final stack with the last
command, take the following steps to see the comparisons properly:
  1. Click on the stack image so it is selected.
  2. Go to the “Frame” menu, then the “Lock” item, then activate “Scale
     and Limits”.
  3. Scroll your mouse or touchpad to zoom into the image.

You clearly see that the stacked image is deeper and that there is no
Moiré pattern, while you have slightly _improved_ the spatial resolution
of the output compared to the input.  In case you want the stack to have
the original pixel resolution, you just need one more warp:

     $ astwarp jplus-stack.fits --cdelt=$cdelt -ojplus-stack-origres.fits

   For optimal results, the oversampling should be determined by the
dithering pattern of the observation: For example if you only have two
dither points, you want the pixels with maximum value in the ‘MAX-FRAC’
image of one exposure to fall on those with a minimum value in the other
exposure.  Ideally, many more dither points should be chosen when you
are planning your observation (not just for the Moiré pattern, but also
for all the other reasons mentioned above).  Based on the dithering
pattern, you want to select the increased resolution such that the
maximum ‘MAX-FRAC’ values fall on every different pixel of the output
grid for each exposure.  Note that this discussion is on small shifts
between pointings (dithers), not large ones like offsets); see *note
Pointing pattern design::.

   ---------- Footnotes ----------

   (1) E.g.,
<https://www.stsci.edu/hst/instrumentation/wfc3/proposing/dithering-strategies>


File: gnuastro.info,  Node: Installation,  Next: Common program behavior,  Prev: Tutorials,  Up: Top

3 Installation
**************

The latest released version of Gnuastro source code is always available
at the following URL:

   <http://ftpmirror.gnu.org/gnuastro/gnuastro-latest.tar.gz>

*note Quick start:: describes the commands necessary to configure,
build, and install Gnuastro on your system.  This chapter will be useful
in cases where the simple procedure above is not sufficient, for
example, your system lacks a mandatory/optional dependency (in other
words, you cannot pass the ‘$ ./configure’ step), or you want greater
customization, or you want to build and install Gnuastro from other
random points in its history, or you want a higher level of control on
the installation.  Thus if you were happy with downloading the tarball
and following *note Quick start::, then you can safely ignore this
chapter and come back to it in the future if you need more
customization.

   *note Dependencies:: describes the mandatory, optional and
bootstrapping dependencies of Gnuastro.  Only the first group are
required/mandatory when you are building Gnuastro using a tarball (see
*note Release tarball::), they are very basic and low-level tools used
in most astronomical software, so you might already have them installed,
if not they are very easy to install as described for each.  *note
Downloading the source:: discusses the two methods you can obtain the
source code: as a tarball (a significant snapshot in Gnuastro’s
history), or the full history(1).  The latter allows you to build
Gnuastro at any random point in its history (for example, to get bug
fixes or new features that are not released as a tarball yet).

   The building and installation of Gnuastro is heavily customizable, to
learn more about them, see *note Build and install::.  This section is
essentially a thorough explanation of the steps in *note Quick start::.
It discusses ways you can influence the building and installation.  If
you encounter any problems in the installation process, it is probably
already explained in *note Known issues::.  In *note Other useful
software:: the installation and usage of some other free software that
are not directly required by Gnuastro but might be useful in conjunction
with it is discussed.

* Menu:

* Dependencies::                Necessary packages for Gnuastro.
* Downloading the source::      Ways to download the source code.
* Build and install::           Configure, build and install Gnuastro.

   ---------- Footnotes ----------

   (1) *note Bootstrapping dependencies:: are required if you clone the
full history.


File: gnuastro.info,  Node: Dependencies,  Next: Downloading the source,  Prev: Installation,  Up: Installation

3.1 Dependencies
================

A minimal set of dependencies are mandatory for building Gnuastro from
the standard tarball release.  If they are not present you cannot pass
Gnuastro’s configuration step.  The mandatory dependencies are therefore
very basic (low-level) tools which are easy to obtain, build and
install, see *note Mandatory dependencies:: for a full discussion.

   If you have the packages of *note Optional dependencies::, Gnuastro
will have additional functionality (for example, converting FITS images
to JPEG or PDF). If you are installing from a tarball as explained in
*note Quick start::, you can stop reading after this section.  If you
are cloning the version controlled source (see *note Version controlled
source::), an additional bootstrapping step is required before
configuration and its dependencies are explained in *note Bootstrapping
dependencies::.

   Your operating system’s package manager is an easy and convenient way
to download and install the dependencies that are already pre-built for
your operating system.  In *note Dependencies from package managers::,
we will list some common operating system package manager commands to
install the optional and mandatory dependencies.

* Menu:

* Mandatory dependencies::      Gnuastro will not install without these.
* Optional dependencies::       Adding more functionality.
* Bootstrapping dependencies::  If you have the version controlled source.
* Dependencies from package managers::  Installing from OS package managers.


File: gnuastro.info,  Node: Mandatory dependencies,  Next: Optional dependencies,  Prev: Dependencies,  Up: Dependencies

3.1.1 Mandatory dependencies
----------------------------

The mandatory Gnuastro dependencies are very basic and low-level tools.
They all follow the same basic GNU based build system (like that shown
in *note Quick start::), so even if you do not have them, installing
them should be pretty straightforward.  In this section we explain each
program and any specific note that might be necessary in the
installation.

* Menu:

* GNU Scientific Library::      Installing GSL.
* CFITSIO::                     C interface to the FITS standard.
* WCSLIB::                      C interface to the WCS standard of FITS.


File: gnuastro.info,  Node: GNU Scientific Library,  Next: CFITSIO,  Prev: Mandatory dependencies,  Up: Mandatory dependencies

3.1.1.1 GNU Scientific Library
..............................

The GNU Scientific Library (http://www.gnu.org/software/gsl/), or GSL,
is a large collection of functions that are very useful in scientific
applications, for example, integration, random number generation, and
Fast Fourier Transform among many others.  To download and install GSL
from source, you can run the following commands.

     $ wget http://ftpmirror.gnu.org/gsl/gsl-latest.tar.gz
     $ tar xf gsl-latest.tar.gz
     $ cd gsl-X.X                     # Replace X.X with version number.
     $ ./configure CFLAGS="$CFLAGS -g0 -O3"
     $ make -j8                       # Replace 8 with no. CPU threads.
     $ make check
     $ sudo make install


File: gnuastro.info,  Node: CFITSIO,  Next: WCSLIB,  Prev: GNU Scientific Library,  Up: Mandatory dependencies

3.1.1.2 CFITSIO
...............

CFITSIO (http://heasarc.gsfc.nasa.gov/fitsio/) is the closest you can
get to the pixels in a FITS image while remaining faithful to the FITS
standard (http://fits.gsfc.nasa.gov/fits_standard.html).  It is written
by William Pence, the principal author of the FITS standard(1), and is
regularly updated.  Setting the definitions for all other software
packages using FITS images.

   Some GNU/Linux distributions have CFITSIO in their package managers,
if it is available and updated, you can use it.  One problem that might
occur is that CFITSIO might not be configured with the
‘--enable-reentrant’ option by the distribution.  This option allows
CFITSIO to open a file in multiple threads, it can thus provide great
speed improvements.  If CFITSIO was not configured with this option, any
program which needs this capability will warn you and abort when you ask
for multiple threads (see *note Multi-threaded operations::).

   To install CFITSIO from source, we strongly recommend that you have a
look through Chapter 2 (Creating the CFITSIO library) of the CFITSIO
manual and understand the options you can pass to ‘$ ./configure’ (they
are not too much).  This is a very basic package for most astronomical
software and it is best that you configure it nicely with your system.
Once you download the source and unpack it, the following configure
script should be enough for most purposes.  Do Not forget to read
chapter two of the manual though, for example, the second option is only
for 64bit systems.  The manual also explains how to check if it has been
installed correctly.

   CFITSIO comes with two executable files called ‘fpack’ and ‘funpack’.
From their manual: they “are standalone programs for compressing and
uncompressing images and tables that are stored in the FITS (Flexible
Image Transport System) data format.  They are analogous to the gzip and
gunzip compression programs except that they are optimized for the types
of astronomical images that are often stored in FITS format”.  The
commands below will compile and install them on your system along with
CFITSIO. They are not essential for Gnuastro, since they are just
wrappers for functions within CFITSIO, but they can come in handy.  The
‘make utils’ command is only available for versions above 3.39, it will
build these executable files along with several other executable test
files which are deleted in the following commands before the
installation (otherwise the test files will also be installed).

   The commands necessary to download the source, decompress, build and
install CFITSIO from source are described below.

     $ urlbase=http://heasarc.gsfc.nasa.gov/FTP/software/fitsio/c
     $ wget $urlbase/cfitsio_latest.tar.gz
     $ tar xf cfitsio_latest.tar.gz
     $ cd cfitsio-X.XX                   # Replace X.XX with version
     $ ./configure --prefix=/usr/local --enable-sse2 --enable-reentrant \
                   CFLAGS="$CFLAGS -g0 -O3"
     $ make
     $ make utils
     $ ./testprog > testprog.lis         # See below if this has an error
     $ diff testprog.lis testprog.out    # Should have no output
     $ cmp testprog.fit testprog.std     # Should have no output
     $ rm cookbook fitscopy imcopy smem speed testprog
     $ sudo make install

   In the ‘./testprog > testprog.lis’ step, you may confront an error,
complaining that it cannot find ‘libcfitsio.so.AAA’ (where ‘AAA’ is an
integer).  This is the library that you just built and have not yet
installed.  But unfortunately some versions of CFITSIO do not account
for this on some OSs.  To fix the problem, you need to tell your OS to
also look into current CFITSIO build directory with the first command
below, afterwards, the problematic command (second below) should run
properly.

     $ export LD_LIBRARY_PATH="$(pwd):$LD_LIBRARY_PATH"
     $ ./testprog > testprog.lis

   Recall that the modification above is ONLY NECESSARY FOR THIS STEP.
_Do Not_ put the ‘LD_LIBRARY_PATH’ modification command in a permanent
place (like your bash startup file).  After installing CFITSIO, close
your terminal and continue working on a new terminal (so
‘LD_LIBRARY_PATH’ has its default value).  For more on
‘LD_LIBRARY_PATH’, see *note Installation directory::.

   ---------- Footnotes ----------

   (1) Pence, W.D. et al.  Definition of the Flexible Image Transport
System (FITS), version 3.0.  (2010) Astronomy and Astrophysics, Volume
524, id.A42, 40 pp.


File: gnuastro.info,  Node: WCSLIB,  Prev: CFITSIO,  Up: Mandatory dependencies

3.1.1.3 WCSLIB
..............

WCSLIB (http://www.atnf.csiro.au/people/mcalabre/WCS/) is written and
maintained by one of the authors of the World Coordinate System (WCS)
definition in the FITS standard
(http://fits.gsfc.nasa.gov/fits_standard.html)(1), Mark Calabretta.  It
might be already built and ready in your distribution’s package
management system.  However, here the installation from source is
explained, for the advantages of installation from source please see
*note Mandatory dependencies::.  To install WCSLIB you will need to have
CFITSIO already installed, see *note CFITSIO::.

   WCSLIB also has plotting capabilities which use PGPLOT (a plotting
library for C). If you wan to use those capabilities in WCSLIB, *note
PGPLOT:: provides the PGPLOT installation instructions.  However PGPLOT
is old(2), so its installation is not easy, there are also many great
modern WCS plotting tools (mostly in written in Python).  Hence, if you
will not be using those plotting functions in WCSLIB, you can configure
it with the ‘--without-pgplot’ option as shown below.

   If you have the cURL library (3) on your system and you installed
CFITSIO version 3.42 or later, you will need to also link with the cURL
library at configure time (through the ‘-lcurl’ option as shown below).
CFITSIO uses the cURL library for its HTTPS (or HTTP Secure(4)) support
and if it is present on your system, CFITSIO will depend on it.
Therefore, if ‘./configure’ command below fails (you do not have the
cURL library), then remove this option and rerun it.

   To download, configure, build, check and install WCSLIB from source,
you can follow the steps below.
     ## Download and unpack the source tarball
     $ wget ftp://ftp.atnf.csiro.au/pub/software/wcslib/wcslib.tar.bz2
     $ tar xf wcslib.tar.bz2

     ## In the `cd' command, replace `X.X' with version number.
     $ cd wcslib-X.X

     ## If `./configure' fails, remove `-lcurl' and run again.
     $ ./configure LIBS="-pthread -lcurl -lm" --without-pgplot     \
                   --disable-fortran CFLAGS="$CFLAGS -g0 -O3"
     $ make
     $ make check
     $ sudo make install

   ---------- Footnotes ----------

   (1) Greisen E.W., Calabretta M.R. (2002) Representation of world
coordinates in FITS. Astronomy and Astrophysics, 395, 1061-1075.

   (2) As of early June 2016, its most recent version was uploaded in
February 2001.

   (3) <https://curl.haxx.se>

   (4) <https://en.wikipedia.org/wiki/HTTPS>


File: gnuastro.info,  Node: Optional dependencies,  Next: Bootstrapping dependencies,  Prev: Mandatory dependencies,  Up: Dependencies

3.1.2 Optional dependencies
---------------------------

The libraries listed here are only used for very specific applications,
therefore they are optional and Gnuastro can be built without them (with
only those specific features disabled).  Since these are pretty
low-level tools, they are not too hard to install from source, but you
can also use your operating system’s package manager to easily install
all of them.  For more, see *note Dependencies from package managers::.

   If the ‘./configure’ script cannot find any of these optional
dependencies, it will notify you of the operation(s) you cannot do due
to not having them.  If you continue the build and request an operation
that uses a missing library, Gnuastro’s programs will warn that the
optional library was missing at build-time and abort.  Since Gnuastro
was built without that library, installing the library afterwards will
not help.  The only way is to re-build Gnuastro from scratch (after the
library has been installed).  However, for program dependencies (like
cURL or Ghostscript) things are easier: you can install them after
building Gnuastro also.  This is because libraries are used to build the
internal structure of Gnuastro’s executables.  However, a program
dependency is called by Gnuastro’s programs at run-time and has no
effect on their internal structure.  So if a dependency program becomes
available later, it will be used next time it is requested.

GNU Libtool
     Libtool is a program to simplify managing of the libraries to build
     an executable (a program).  GNU Libtool has some added
     functionality compared to other implementations.  If GNU Libtool is
     not present on your system at configuration time, a warning will be
     printed and *note BuildProgram:: will not be built or installed.
     The configure script will look into your search path (‘PATH’) for
     GNU Libtool through the following executable names: ‘libtool’
     (acceptable only if it is the GNU implementation) or ‘glibtool’.
     See *note Installation directory:: for more on ‘PATH’.

     GNU Libtool (the binary/executable file) is a low-level program
     that is probably already present on your system, and if not, is
     available in your operating system package manager(1).  If you want
     to install GNU Libtool’s latest version from source, please visit
     its web page (https://www.gnu.org/software/libtool/).

     Gnuastro’s tarball is shipped with an internal implementation of
     GNU Libtool.  Even if you have GNU Libtool, Gnuastro’s internal
     implementation is used for the building and installation of
     Gnuastro.  As a result, you can still build, install and use
     Gnuastro even if you do not have GNU Libtool installed on your
     system.  However, this internal Libtool does not get installed.
     Therefore, after Gnuastro’s installation, if you want to use *note
     BuildProgram:: to compile and link your own C source code which
     uses the *note Gnuastro library::, you need to have GNU Libtool
     available on your system (independent of Gnuastro).  See *note
     Review of library fundamentals:: to learn more about libraries.

GNU Make extension headers
     GNU Make is a workflow management system that can be used to run a
     series of commands in a specific order, and in parallel if you
     want.  GNU Make offers special features to extend it with custom
     functions within a dynamic library.  They are defined in the
     ‘gnumake.h’ header.  If ‘gnumake.h’ can be found on your system at
     configuration time, Gnuastro will build a custom library that GNU
     Make can use for extended functionality in (astronomical) data
     analysis scenarios.

libgit2
     Git is one of the most common version control systems (see *note
     Version controlled source::).  When ‘libgit2’ is present, and
     Gnuastro’s programs are run within a version controlled directory,
     outputs will contain the version number of the working directory’s
     repository for future reproducibility.  See the ‘COMMIT’ keyword
     header in *note Output FITS files:: for a discussion.

libjpeg
     libjpeg is only used by ConvertType to read from and write to JPEG
     images, see *note Recognized file formats::.  libjpeg
     (http://www.ijg.org/) is a very basic library that provides tools
     to read and write JPEG images, most Unix-like graphic programs and
     libraries use it.  Therefore you most probably already have it
     installed.  libjpeg-turbo (http://libjpeg-turbo.virtualgl.org/) is
     an alternative to libjpeg.  It uses Single instruction, multiple
     data (SIMD) instructions for ARM based systems that significantly
     decreases the processing time of JPEG compression and decompression
     algorithms.

libtiff
     libtiff is used by ConvertType and the libraries to read TIFF
     images, see *note Recognized file formats::.  libtiff
     (http://www.simplesystems.org/libtiff/) is a very basic library
     that provides tools to read and write TIFF images, most Unix-like
     operating system graphic programs and libraries use it.  Therefore
     even if you do not have it installed, it must be easily available
     in your package manager.

cURL
     cURL’s executable (‘curl’) is called by *note Query:: for
     submitting queries to remote datasets and retrieving the results.
     It is not necessary for the build of Gnuastro from source (only a
     warning will be printed if it cannot be found at configure time),
     so if you do not have it at build-time there is no problem.  Just
     be sure to have it when you run ‘astquery’, otherwise you’ll get an
     error about not finding ‘curl’.

GPL Ghostscript
     GPL Ghostscript’s executable (‘gs’) is called by ConvertType to
     compile a PDF file from a source PostScript file, see *note
     ConvertType::.  Therefore its headers (and libraries) are not
     needed.

Python3 with Numpy
     Python is a high-level programming language and Numpy is the most
     commonly used library within Python to add multi-dimensional arrays
     and matrices.  If you configure Gnuastro with ‘--with-python’ _and_
     version 3 of Python is available with a corresponding Numpy
     Library, Gnuastro’s library will be built with some Python-related
     helper functions.  Python wrappers for Gnuastro’s library (for
     example, ‘pyGnuastro’) can use these functions when being built
     from source.  For more on Gnuastro’s Python helper functions, see
     *note Python interface::.

     This Python interface is only relevant if you want to build the
     Python wrappers (like ‘pyGnuastro’) from source.  If you install
     the Gnuastro Python wrapper from a pre-built repository like PyPI,
     this feature of your Gnuastro library won’t be used.  Pre-built
     libraries contain the full Gnuastro library that they need within
     them (you don’t even need to have Gnuastro at all!).

     *Can’t find the Python3 and Numpy of a virtual environment:* make
     sure to set the ‘$PYTHON’ variable to point to the ‘python3’
     command of the virtual environment before running ‘./configure’.
     Note that you don’t need to activate the virtual env, just point
     ‘PYTHON’ to its Python3 executable, like the example below:

          $ python3 -m venv test-env    # Setting up the virtual env.
          $ export PYTHON="$(pwd)/test-env/bin/python3"
          $ ./configure                 # Gnuastro's configure script.

SAO DS9
     SAO DS9 (‘ds9’) is a visualization tool for FITS images.
     Gnuastro’s ‘astscript-fits-view’ program calls DS9 to visualize
     FITS images.  We have a full appendix on it and how to install it
     in *note SAO DS9::.  Since it is a run-time dependency, it can be
     installed at any later time (after building and installing
     Gnuastro).

TOPCAT
     TOPCAT (‘topcat’) is a visualization tool for astronomical tables
     (most commonly: plotting).  Gnuastro’s ‘astscript-fits-view’
     program calls TOPCAT it to visualize tables.  We have a full
     appendix on it and how to install it in *note TOPCAT::.  Since it
     is a run-time dependency, it can be installed at any later time
     (after building and installing Gnuastro).

   ---------- Footnotes ----------

   (1) Note that we want the binary/executable Libtool program which can
be run on the command-line.  In Debian-based operating systems which
separate various parts of a package, you want want ‘libtool-bin’, the
‘libtool’ package will not contain the executable program.


File: gnuastro.info,  Node: Bootstrapping dependencies,  Next: Dependencies from package managers,  Prev: Optional dependencies,  Up: Dependencies

3.1.3 Bootstrapping dependencies
--------------------------------

Bootstrapping is only necessary if you have decided to obtain the full
version controlled history of Gnuastro, see *note Version controlled
source:: and *note Bootstrapping::.  Using the version controlled source
enables you to always be up to date with the most recent development
work of Gnuastro (bug fixes, new functionalities, improved algorithms,
etc.).  If you have downloaded a tarball (see *note Downloading the
source::), then you can ignore this subsection.

   To successfully run the bootstrapping process, there are some
additional dependencies to those discussed in the previous subsections.
These are low level tools that are used by a large collection of
Unix-like operating systems programs, therefore they are most probably
already available in your system.  If they are not already installed,
you should be able to easily find them in any GNU/Linux distribution
package management system (‘apt-get’, ‘yum’, ‘pacman’, etc.).  The short
names in parenthesis in ‘typewriter’ font after the package name can be
used to search for them in your package manager.  For the GNU
Portability Library, GNU Autoconf Archive and TeX Live, it is
recommended to use the instructions here, not your operating system’s
package manager.

GNU Portability Library (Gnulib)
     To ensure portability for a wider range of operating systems (those
     that do not include GNU C library, namely glibc), Gnuastro depends
     on the GNU portability library, or Gnulib.  Gnulib keeps a copy of
     all the functions in glibc, implemented (as much as possible) to be
     portable to other operating systems.  The ‘bootstrap’ script can
     automatically clone Gnulib (as a ‘gnulib/’ directory inside
     Gnuastro), however, as described in *note Bootstrapping:: this is
     not recommended.

     The recommended way to bootstrap Gnuastro is to first clone Gnulib
     and the Autoconf archives (see below) into a local directory
     outside of Gnuastro.  Let’s call it ‘DEVDIR’(1) (which you can set
     to any directory; preferentially where you keep your other
     development projects).  Currently in Gnuastro, both Gnulib and
     Autoconf archives have to be cloned in the same top directory(2)
     like the case here(3):

          $ DEVDIR=/home/yourname/Development  ## Select any location.
          $ mkdir $DEVDIR                      ## If it doesn't exist!
          $ cd $DEVDIR
          $ git clone https://git.sv.gnu.org/git/gnulib.git
          $ git clone https://git.sv.gnu.org/git/autoconf-archive.git

     Gnulib is a source-based dependency of Gnuastro’s bootstrapping
     process, so simply having it is enough on your computer, there is
     no need to install, and thus check anything.

     You now have the full version controlled source of these two
     repositories in separate directories.  Both these packages are
     regularly updated, so every once in a while, you can run ‘$ git
     pull’ within them to get any possible updates.

GNU Automake (‘automake’)
     GNU Automake will build the ‘Makefile.in’ files in each
     sub-directory using the (hand-written) ‘Makefile.am’ files.  The
     ‘Makefile.in’s are subsequently used to generate the ‘Makefile’s
     when the user runs ‘./configure’ before building.

     To check that you have a working GNU Automake in your system, you
     can try this command:

          $ automake --version

GNU Autoconf (‘autoconf’)
     GNU Autoconf will build the ‘configure’ script using the
     configurations we have defined (hand-written) in ‘configure.ac’.

     To check that you have a working GNU Autoconf in your system, you
     can try this command:

          $ autoconf --version

GNU Autoconf Archive
     These are a large collection of tests that can be called to run at
     ‘./configure’ time.  See the explanation under GNU Portability
     Library (Gnulib) above for instructions on obtaining it and keeping
     it up to date.

     GNU Autoconf Archive is a source-based dependency of Gnuastro’s
     bootstrapping process, so simply having it is enough on your
     computer, there is no need to install, and thus check anything.
     Just do not forget that it has to be in the same directory as
     Gnulib (described above).

GNU Texinfo (‘texinfo’)
     GNU Texinfo is the tool that formats this manual into the various
     output formats.  To bootstrap Gnuastro you need all of Texinfo’s
     command-line programs.  However, some operating systems package
     them separately, for example, in Fedora, ‘makeinfo’ is packaged in
     the ‘texinfo-tex’ package.

     To check that you have a working GNU Texinfo in your system, you
     can try this command:

          $ makeinfo --version

GNU Libtool (‘libtool’)
     GNU Libtool is in charge of building all the libraries in Gnuastro.
     The libraries contain functions that are used by more than one
     program and are installed for use in other programs.  They are thus
     put in a separate directory (‘lib/’).

     To check that you have a working GNU Libtool in your system, you
     can try this command (and from the output, make sure it is GNU’s
     libtool)

          $ libtool --version

GNU help2man (‘help2man’)
     GNU help2man is used to convert the output of the ‘--help’ option
     (*note --help::) to the traditional Man page (*note Man pages::).

     To check that you have a working GNU Help2man in your system, you
     can try this command:

          $ help2man --version

LaTeX and some TeX packages
     Some of the figures in this book are built by LaTeX (using the
     PGF/TikZ package).  The LaTeX source for those figures is version
     controlled for easy maintenance not the actual figures.  So the
     ‘./boostrap’ script will run LaTeX to build the figures.  The best
     way to install LaTeX and all the necessary packages is through TeX
     live (https://www.tug.org/texlive/) which is a package manager for
     TeX related tools that is independent of any operating system.  It
     is thus preferred to the TeX Live versions distributed by your
     operating system.

     To install TeX Live, go to the web page and download the
     appropriate installer by following the “download” link.  Note that
     by default the full package repository will be downloaded and
     installed (around 4 Gigabytes) which can take _very_ long to
     download and to update later.  However, most packages are not
     needed by everyone, it is easier, faster and better to install only
     the “Basic scheme” (consisting of only the most basic TeX and LaTeX
     packages, which is less than 200 Mega bytes)(4).

     After the installation, be sure to set the environment variables as
     suggested in the end of the outputs.  Any time you confront (need)
     a package you do not have, simply install it with a command like
     below (similar to how you install software from your operating
     system’s package manager)(5).  To install all the necessary TeX
     packages for a successful Gnuastro bootstrap, run this command:

          $ sudo su
          # tlmgr install epsf jknapltx caption biblatex biber iftex \
                          etoolbox logreq xstring xkeyval pgf ms     \
                          xcolor pgfplots times rsfs ps2eps epspdf

     To check that you have a working LaTeX executable in your system,
     you can try this command (this just checks if LaTeX exists, as
     described above, if you have a missing package, you can easily
     identify it from the output and install it with ‘tlmgr’):

          $ latex --version

ImageMagick (‘imagemagick’)
     ImageMagick is a wonderful and robust program for image
     manipulation on the command-line.  ‘bootstrap’ uses it to convert
     the book images into the formats necessary for the various book
     formats.

     Since ImageMagick version 7, it is necessary to edit the policy
     file (‘/etc/ImageMagick-7/policy.xml’) to have the following line
     (it maybe present, but commented, in this case un-comment it):

          <policy domain="coder" rights="read|write" pattern="{PS,PDF,XPS}"/>

     If the following line is present, it is also necessary to
     comment/remove it.

          <policy domain="delegate" rights="none" pattern="gs" />

     To learn more about the ImageMagick security policy please see:
     <https://imagemagick.org/script/security-policy.php>.

     To check that you have a working ImageMagick in your system, you
     can try this command:

          $ convert --version

   ---------- Footnotes ----------

   (1) If you are not a developer in Gnulib or Autoconf archives,
‘DEVDIR’ can be a directory that you do not backup.  In this way the
large number of files in these projects will not slow down your backup
process or take bandwidth (if you backup to a remote server).

   (2) If you already have the Autoconf archives in a separate
directory, or cannot clone it in the same directory as Gnulib, or you
have it with another directory name (not ‘autoconf-archive/’), you can
follow this short step.  Set ‘AUTOCONFARCHIVES’ to your desired address.
Then define a symbolic link in ‘DEVDIR’ with the following command so
Gnuastro’s bootstrap script can find it:
‘$ ln -s $AUTOCONFARCHIVES $DEVDIR/autoconf-archive’.

   (3) If your internet connection is active, but Git complains about
the network, it might be due to your network setup not recognizing the
git protocol.  In that case use the following URL for the HTTP protocol
instead (for Autoconf archives, replace the name):
‘http://git.sv.gnu.org/r/gnulib.git’

   (4) You can also download the DVD iso file at a later time to keep as
a backup for when you do not have internet connection if you need a
package.

   (5) After running TeX, or LaTeX, you might get a warning complaining
about a ‘missingfile’.  Run ‘‘tlmgr info missingfile’’ to see the
package(s) containing that file which you can install.


File: gnuastro.info,  Node: Dependencies from package managers,  Prev: Bootstrapping dependencies,  Up: Dependencies

3.1.4 Dependencies from package managers
----------------------------------------

The most basic way to install a package on your system is to build the
packages from source yourself.  Alternatively, you can use your
operating system’s package manager to download pre-compiled files and
install them.  The latter choice is easier and faster.  However, we
recommend that you build the *note Mandatory dependencies:: yourself
from source (all necessary commands and links are given in the
respective section).  Here are some basic reasons behind this
recommendation.

  1. Your operating system’s pre-built software might not be the most
     recent release.  For example, Gnuastro itself is also packaged in
     some package managers.  For the list see:
     <https://repology.org/project/gnuastro/versions>.  You will notice
     that Gnuastro’s version in some operating systems is more than 10
     versions old!  It is the same for all the dependencies of Gnuastro.

  2. For each package, Gnuastro might preform better (or require)
     certain configuration options that your distribution’s package
     managers did not add for you.  If present, these configuration
     options are explained during the installation of each in the
     sections below (for example, in *note CFITSIO::).  When the proper
     configuration has not been set, the programs should complain and
     inform you.

  3. For the libraries, they might separate the binary file from the
     header files which can cause confusion, see *note Known issues::.

  4. Like any other tool, the science you derive from Gnuastro’s tools
     highly depend on these lower level dependencies, so generally it is
     much better to have a close connection with them.  By reading their
     manuals, installing them and staying up to date with changes/bugs
     in them, your scientific results and understanding (of what is
     going on, and thus how you interpret your scientific results) will
     also correspondingly improve.

   Based on your package manager, you can use any of the following
commands to install the mandatory and optional dependencies.  If your
package manager is not included in the list below, please send us the
respective command, so we add it.  For better archivability and
compression ratios, Gnuastro’s recommended tarball compression format is
with the Lzip (http://lzip.nongnu.org/lzip.html) program, see *note
Release tarball::.  Therefore, the package manager commands below also
contain Lzip.

‘apt-get’ (Debian-based OSs: Debian, Ubuntu, Linux Mint, etc.)
     Debian (https://en.wikipedia.org/wiki/Debian) is one of the oldest
     GNU/Linux distributions(1).  It thus has a very extended user
     community and a robust internal structure and standards.  All of it
     is free software and based on the work of volunteers around the
     world.  Many distributions are thus derived from it, for example,
     Ubuntu and Linux Mint.  This arguably makes Debian-based OSs the
     largest, and most used, class of GNU/Linux distributions.  All of
     them use Debian’s Advanced Packaging Tool (APT, for example,
     ‘apt-get’) for managing packages.

     Mandatory dependencies
          Without these, Gnuastro cannot be built, they are necessary
          for input/output and low-level mathematics (see *note
          Mandatory dependencies::)!
               $ sudo apt-get install libgsl-dev libcfitsio-dev \
                                      wcslib-dev

     Optional dependencies
          If present, these libraries can be used in Gnuastro’s build
          for extra features, see *note Optional dependencies::.
               $ sudo apt-get install ghostscript libtool-bin \
                                      libjpeg-dev libtiff-dev \
                                      libgit2-dev curl lzip

     Programs to view FITS images or tables
          These are not used in Gnuastro’s build.  They can just help in
          viewing the inputs/outputs independent of Gnuastro!
               $ sudo apt-get install saods9 topcat

     Gnuastro is packaged (https://tracker.debian.org/pkg/gnuastro) in
     Debian (and thus some of its derivate operating systems).  Just
     make sure it is the most recent version.

‘dnf’
‘yum’ (Red Hat-based OSs: Red Hat, Fedora, CentOS, Scientific Linux, etc.)
     Red Hat Enterprise Linux (https://en.wikipedia.org/wiki/Red_Hat)
     (RHEL) is released by Red Hat Inc.  RHEL requires paid
     subscriptions for use of its binaries and support.  But since it is
     free software, many other teams use its code to spin-off their own
     distributions based on RHEL. Red Hat-based GNU/Linux distributions
     initially used the “Yellowdog Updated, Modifier” (YUM) package
     manager, which has been replaced by “Dandified yum” (DNF). If the
     latter is not available on your system, you can use ‘yum’ instead
     of ‘dnf’ in the command below.

     Mandatory dependencies
          Without these, Gnuastro cannot be built, they are necessary
          for input/output and low-level mathematics (see *note
          Mandatory dependencies::)!
               $ sudo dnf install gsl-devel cfitsio-devel \
                                  wcslib-devel

     Optional dependencies
          If present, these libraries can be used in Gnuastro’s build
          for extra features, see *note Optional dependencies::.
               $ sudo dnf install ghostscript libtool \
                                  libjpeg-devel libtiff-devel \
                                  libgit2-devel lzip curl

     Programs to view FITS images or tables
          These are not used in Gnuastro’s build.  They can just help in
          viewing the inputs/outputs independent of Gnuastro!
               $ sudo dnf install saods9 topcat

‘brew’ (macOS)
     macOS (https://en.wikipedia.org/wiki/MacOS) is the operating system
     used on Apple devices.  macOS does not come with a package manager
     pre-installed, but several widely used, third-party package
     managers exist, such as Homebrew or MacPorts.  Both are free
     software.  Currently we have only tested Gnuastro’s installation
     with Homebrew as described below.  If not already installed, first
     obtain Homebrew by following the instructions at <https://brew.sh>.

     Mandatory dependencies
          Without these, Gnuastro cannot be built, they are necessary
          for input/output and low-level mathematics (see *note
          Mandatory dependencies::)!

          Homebrew manages packages in different ‘taps’.  To install
          WCSLIB via Homebrew you will need to ‘tap’ into
          ‘brewsci/science’ first (the tap may change in the future, but
          can be found by calling ‘brew search wcslib’).
               $ brew tap brewsci/science
               $ brew install wcslib gsl cfitsio

     Optional dependencies
          If present, these libraries can be used in Gnuastro’s build
          for extra features, see *note Optional dependencies::.
               $ brew install ghostscript libtool libjpeg \
                              libtiff libgit2 curl lzip

     Programs to view FITS images or tables
          These are not used in Gnuastro’s build.  They can just help in
          viewing the inputs/outputs independent of Gnuastro!
               $ brew install saoimageds9 topcat

‘pacman’ (Arch Linux)
     Arch Linux (https://en.wikipedia.org/wiki/Arch_Linux) is a smaller
     GNU/Linux distribution, which follows the KISS principle (“keep it
     simple, stupid”) as a general guideline.  It “focuses on elegance,
     code correctness, minimalism and simplicity, and expects the user
     to be willing to make some effort to understand the system’s
     operation”.  Arch GNU/Linux uses “Package manager” (Pacman) to
     manage its packages/components.

     Mandatory dependencies
          Without these, Gnuastro cannot be built, they are necessary
          for input/output and low-level mathematics (see *note
          Mandatory dependencies::)!
               $ sudo pacman -S gsl cfitsio wcslib

     Optional dependencies
          If present, these libraries can be used in Gnuastro’s build
          for extra features, see *note Optional dependencies::.
               $ sudo pacman -S ghostscript libtool libjpeg \
                                libtiff libgit2 curl lzip

     Programs to view FITS images or tables
          These are not used in Gnuastro’s build.  They can just help in
          viewing the inputs/outputs independent of Gnuastro!

          SAO DS9 and TOPCAT are not available in the standard Arch
          GNU/Linux repositories.  However, installing and using both is
          very easy from their own web pages, as described in *note SAO
          DS9:: and *note TOPCAT::.

‘zypper’ (openSUSE and SUSE Linux Enterprise Server)
     SUSE Linux Enterprise Server(2) (SLES) is the commercial offering
     which shares code and tools.  Many additional packages are offered
     in the Build Service(3).  openSUSE and SLES use ‘zypper’ (cli) and
     YaST (GUI) for managing repositories and packages.

     Configuration
          When building Gnuastro, run the configure script with the
          following ‘CPPFLAGS’ environment variable:

               $ ./configure CPPFLAGS="-I/usr/include/cfitsio"

     Mandatory dependencies
          Without these, Gnuastro cannot be built, they are necessary
          for input/output and low-level mathematics (see *note
          Mandatory dependencies::)!
               $ sudo zypper install gsl-devel cfitsio-devel \
                                     wcslib-devel

     Optional dependencies
          If present, these libraries can be used in Gnuastro’s build
          for extra features, see *note Optional dependencies::.
               $ sudo zypper install ghostscript_any libtool \
                                     pkgconfig libcurl-devel \
                                     libgit2-devel \
                                     libjpeg62-devel \
                                     libtiff-devel curl

     Programs to view FITS images or tables
          These are not used in Gnuastro’s build.  They can just help in
          viewing the inputs/outputs independent of Gnuastro!
               $ sudo zypper install ds9 topcat

   Usually, when libraries are installed by operating system package
managers, there should be no problems when configuring and building
other programs from source (that depend on the libraries: Gnuastro in
this case).  However, in some special conditions, problems may pop-up
during the configuration, building, or checking/running any of
Gnuastro’s programs.  The most common of such problems and their
solution are discussed below.

*Not finding library during configuration:* If a library is installed,
but during Gnuastro’s ‘configure’ step the library is not found, then
configure Gnuastro like the command below (correcting ‘/path/to/lib’).
For more, see *note Known issues:: and *note Installation directory::.
     $ ./configure LDFLAGS="-L/path/to/lib"

*Not finding header (.h) files while building:* If a library is
installed, but during Gnuastro’s ‘make’ step, the library’s header (file
with a ‘.h’ suffix) is not found, then configure Gnuastro like the
command below (correcting ‘/path/to/include’).  For more, see *note
Known issues:: and *note Installation directory::.
     $ ./configure CPPFLAGS="-I/path/to/include"

*Gnuastro’s programs do not run during check or after install:* If a
library is installed, but the programs do not run due to linking
problems, set the ‘LD_LIBRARY_PATH’ variable like below (assuming
Gnuastro is installed in ‘/path/to/installed’).  For more, see *note
Known issues:: and *note Installation directory::.
     $ export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/path/to/installed/lib"

   ---------- Footnotes ----------

   (1) 
<https://en.wikipedia.org/wiki/List_of_Linux_distributions#Debian-based>

   (2) <https://www.suse.com/products/server>

   (3) <https://build.opensuse.org>


File: gnuastro.info,  Node: Downloading the source,  Next: Build and install,  Prev: Dependencies,  Up: Installation

3.2 Downloading the source
==========================

Gnuastro’s source code can be downloaded in two ways.  As a tarball,
ready to be configured and installed on your system (as described in
*note Quick start::), see *note Release tarball::.  If you want official
releases of stable versions this is the best, easiest and most common
option.  Alternatively, you can clone the version controlled history of
Gnuastro, run one extra bootstrapping step and then follow the same
steps as the tarball.  This will give you access to all the most recent
work that will be included in the next release along with the full
project history.  The process is thoroughly introduced in *note Version
controlled source::.

* Menu:

* Release tarball::             Download a stable official release.
* Version controlled source::   Get and use the version controlled source.


File: gnuastro.info,  Node: Release tarball,  Next: Version controlled source,  Prev: Downloading the source,  Up: Downloading the source

3.2.1 Release tarball
---------------------

A release tarball (commonly compressed) is the most common way of
obtaining free and open source software.  A tarball is a snapshot of one
particular moment in the Gnuastro development history along with all the
necessary files to configure, build, and install Gnuastro easily (see
*note Quick start::).  It is very straightforward and needs the least
set of dependencies (see *note Mandatory dependencies::).  Gnuastro has
tarballs for official stable releases and pre-releases for testing.  See
*note Version numbering:: for more on the two types of releases and the
formats of the version numbers.  The URLs for each type of release are
given below.

Official stable releases (<http://ftp.gnu.org/gnu/gnuastro>):
     This URL hosts the official stable releases of Gnuastro.  Always
     use the most recent version (see *note Version numbering::).  By
     clicking on the “Last modified” title of the second column, the
     files will be sorted by their date which you can also use to find
     the latest version.  It is recommended to use a mirror to download
     these tarballs, please visit <http://ftpmirror.gnu.org/gnuastro/>
     and see below.

Pre-release tarballs (<http://alpha.gnu.org/gnu/gnuastro>):
     This URL contains unofficial pre-release versions of Gnuastro.  The
     pre-release versions of Gnuastro here are for enthusiasts to try
     out before an official release.  If there are problems, or bugs
     then the testers will inform the developers to fix before the next
     official release.  See *note Version numbering:: to understand how
     the version numbers here are formatted.  If you want to remain even
     more up-to-date with the developing activities, please clone the
     version controlled source as described in *note Version controlled
     source::.

   Gnuastro’s official/stable tarball is released with two formats: Gzip
(with suffix ‘.tar.gz’) and Lzip (with suffix ‘.tar.lz’).  The
pre-release tarballs (after version 0.3) are released only as an Lzip
tarball.  Gzip is a very well-known and widely used compression program
created by GNU and available in most systems.  However, Lzip provides a
better compression ratio and more robust archival capacity.  For
example, Gnuastro 0.3’s tarball was 2.9MB and 4.3MB with Lzip and Gzip
respectively, see the Lzip web page
(http://www.nongnu.org/lzip/lzip.html) for more.  Lzip might not be
pre-installed in your operating system, if so, installing it from your
operating system’s package manager or from source is very easy and fast
(it is a very small program).

   The GNU FTP server is mirrored (has backups) in various locations on
the globe (<http://www.gnu.org/order/ftp.html>).  You can use the
closest mirror to your location for a more faster download.  Note that
only some mirrors keep track of the pre-release (alpha) tarballs.  Also
note that if you want to download immediately after and announcement
(see *note Announcements::), the mirrors might need some time to
synchronize with the main GNU FTP server.


File: gnuastro.info,  Node: Version controlled source,  Prev: Release tarball,  Up: Downloading the source

3.2.2 Version controlled source
-------------------------------

The publicly distributed Gnuastro tarball (for example,
‘gnuastro-X.X.tar.gz’) does not contain the revision history, it is only
a snapshot of the source code at one significant instant of Gnuastro’s
history (specified by the version number, see *note Version
numbering::), ready to be configured and built.  To be able to develop
successfully, the revision history of the code can be very useful to
track when something was added or changed, also some updates that are
not yet officially released might be in it.

   We use Git for the version control of Gnuastro.  For those who are
not familiar with it, we recommend the ProGit book
(https://git-scm.com/book/en).  The whole book is publicly available for
online reading and downloading and does a wonderful job at explaining
the concepts and best practices.

   Let’s assume you want to keep Gnuastro in the ‘TOPGNUASTRO’ directory
(can be any directory, change the value below).  The full version
controlled history of Gnuastro can be cloned in ‘TOPGNUASTRO/gnuastro’
by running the following commands(1):

     $ TOPGNUASTRO=/home/yourname/Research/projects/
     $ cd $TOPGNUASTRO
     $ git clone git://git.sv.gnu.org/gnuastro.git

The ‘$TOPGNUASTRO/gnuastro’ directory will contain hand-written (version
controlled) source code for Gnuastro’s programs, libraries, this book
and the tests.  All are divided into sub-directories with standard and
very descriptive names.  The version controlled files in the top cloned
directory are either mainly in capital letters (for example, ‘THANKS’
and ‘README’) or mainly written in small-caps (for example,
‘configure.ac’ and ‘Makefile.am’).  The former are non-programming,
standard writing for human readers containing high-level information
about the whole package.  The latter are instructions to customize the
GNU build system for Gnuastro.  For more on Gnuastro’s source code
structure, please see *note Developing::.  We will not go any deeper
here.

   The cloned Gnuastro source cannot immediately be configured,
compiled, or installed since it only contains hand-written files, not
automatically generated or imported files which do all the hard work of
the build process.  See *note Bootstrapping:: for the process of
generating and importing those files (it is not too hard!).  Once you
have bootstrapped Gnuastro, you can run the standard procedures (in
*note Quick start::).  Very soon after you have cloned it, Gnuastro’s
main ‘master’ branch will be updated on the main repository (since the
developers are actively working on Gnuastro), for the best practices in
keeping your local history in sync with the main repository see *note
Synchronizing::.

* Menu:

* Bootstrapping::               Adding all the automatically generated files.
* Synchronizing::               Keep your local clone up to date.

   ---------- Footnotes ----------

   (1) If your internet connection is active, but Git complains about
the network, it might be due to your network setup not recognizing the
Git protocol.  In that case use the following URL which uses the HTTP
protocol instead: ‘http://git.sv.gnu.org/r/gnuastro.git’


File: gnuastro.info,  Node: Bootstrapping,  Next: Synchronizing,  Prev: Version controlled source,  Up: Version controlled source

3.2.2.1 Bootstrapping
.....................

The version controlled source code lacks the source files that we have
not written or are automatically built.  These automatically generated
files are included in the distributed tarball for each distribution (for
example, ‘gnuastro-X.X.tar.gz’, see *note Version numbering::) and make
it easy to immediately configure, build, and install Gnuastro.  However
from the perspective of version control, they are just bloatware and
sources of confusion (since they are not changed by Gnuastro
developers).

   The process of automatically building and importing necessary files
into the cloned directory is known as _bootstrapping_.  After
bootstrapping is done you are ready to follow the default GNU build
steps that you normally run on the tarball (‘./configure && make’ for
example, described more in *note Quick start::).  Some known issues with
bootstrapping may occur during the process, to see how to fix them,
please see *note Known issues::.

   All the instructions for an automatic bootstrapping are available in
‘bootstrap’ and configured using ‘bootstrap.conf’.  ‘bootstrap’ and
‘COPYING’ (which contains the software copyright notice) are the only
files not written by Gnuastro developers but under version control to
enable simple bootstrapping and legal information on usage immediately
after cloning.  ‘bootstrap.conf’ is maintained by the GNU Portability
Library (Gnulib) and this file is an identical copy, so do not make any
changes in this file since it will be replaced when Gnulib releases an
update.  Make all your changes in ‘bootstrap.conf’.

   The bootstrapping process has its own separate set of dependencies,
the full list is given in *note Bootstrapping dependencies::.  They are
generally very low-level and used by a very large set of commonly used
programs, so they are probably already installed on your system.  The
simplest way to bootstrap Gnuastro is to simply run the bootstrap script
within your cloned Gnuastro directory as shown below.  However, please
read the next paragraph before doing so (see *note Version controlled
source:: for ‘TOPGNUASTRO’).

     $ cd TOPGNUASTRO/gnuastro
     $ ./bootstrap                      # Requires internet connection

   Without any options, ‘bootstrap’ will clone Gnulib within your cloned
Gnuastro directory (‘TOPGNUASTRO/gnuastro/gnulib’) and download the
necessary Autoconf archives macros.  So if you run bootstrap like this,
you will need an internet connection every time you decide to bootstrap.
Also, Gnulib is a large package and cloning it can be slow.  It will
also keep the full Gnulib repository within your Gnuastro repository, so
if another one of your projects also needs Gnulib, and you insist on
running bootstrap like this, you will have two copies.  In case you
regularly backup your important files, Gnulib will also slow down the
backup process.  Therefore while the simple invocation above can be used
with no problem, it is not recommended.  To do better, see the next
paragraph.

   The recommended way to get these two packages is thoroughly discussed
in *note Bootstrapping dependencies:: (in short: clone them in the
separate ‘DEVDIR/’ directory).  The following commands will take you
into the cloned Gnuastro directory and run the ‘bootstrap’ script, while
telling it to copy some files (instead of making symbolic links, with
the ‘--copy’ option, this is not mandatory(1)) and where to look for
Gnulib (with the ‘--gnulib-srcdir’ option).  Please note that the
address given to ‘--gnulib-srcdir’ has to be an absolute address (so do
not use ‘~’ or ‘../’ for example).

     $ cd $TOPGNUASTRO/gnuastro
     $ ./bootstrap --copy --gnulib-srcdir=$DEVDIR/gnulib

   Since Gnulib and Autoconf archives are now available in your local
directories, you do not need an internet connection every time you
decide to remove all un-tracked files and redo the bootstrap (see box
below).  You can also use the same command on any other project that
uses Gnulib.  All the necessary GNU C library functions, Autoconf macros
and Automake inputs are now available along with the book figures.  The
standard GNU build system (*note Quick start::) will do the rest of the
job.

*Undoing the bootstrap:* During the development, it might happen that
you want to remove all the automatically generated and imported files.
In other words, you might want to reverse the bootstrap process.
Fortunately Git has a good program for this job: ‘git clean’.  Run the
following command and every file that is not version controlled will be
removed.

     git clean -fxd

It is best to commit any recent change before running this command.  You
might have created new files since the last commit and if they have not
been committed, they will all be gone forever (using ‘rm’).  To get a
list of the non-version controlled files instead of deleting them, add
the ‘n’ option to ‘git clean’, so it becomes ‘-fxdn’.

   Besides the ‘bootstrap’ and ‘bootstrap.conf’, the ‘bootstrapped/’
directory and ‘README-hacking’ file are also related to the
bootstrapping process.  The former hosts all the imported (bootstrapped)
directories.  Thus, in the version controlled source, it only contains a
‘README’ file, but in the distributed tarball it also contains
sub-directories filled with all bootstrapped files.  ‘README-hacking’
contains a summary of the bootstrapping process discussed in this
section.  It is a necessary reference when you have not built this book
yet.  It is thus not distributed in the Gnuastro tarball.

   ---------- Footnotes ----------

   (1) The ‘--copy’ option is recommended because some backup systems
might do strange things with symbolic links.


File: gnuastro.info,  Node: Synchronizing,  Prev: Bootstrapping,  Up: Version controlled source

3.2.2.2 Synchronizing
.....................

The bootstrapping script (see *note Bootstrapping::) is not regularly
needed: you mainly need it after you have cloned Gnuastro (once) and
whenever you want to re-import the files from Gnulib, or Autoconf
archives(1) (not too common).  However, Gnuastro developers are
constantly working on Gnuastro and are pushing their changes to the
official repository.  Therefore, your local Gnuastro clone will soon be
out-dated.  Gnuastro has two mailing lists dedicated to its developing
activities (see *note Developing mailing lists::).  Subscribing to them
can help you decide when to synchronize with the official repository.

   To pull all the most recent work in Gnuastro, run the following
command from the top Gnuastro directory.  If you do not already have a
built system, ignore ‘make distclean’.  The separate steps are described
in detail afterwards.

     $ make distclean && git pull && autoreconf -f

You can also run the commands separately:

     $ make distclean
     $ git pull
     $ autoreconf -f

   If Gnuastro was already built in this directory, you do not want some
outputs from the previous version being mixed with outputs from the
newly pulled work.  Therefore, the first step is to clean/delete all the
built files with ‘make distclean’.  Fortunately the GNU build system
allows the separation of source and built files (in separate
directories).  This is a great feature to keep your source directory
clean and you can use it to avoid the cleaning step.  Gnuastro comes
with a script with some useful options for this job.  It is useful if
you regularly pull recent changes, see *note Separate build and source
directories::.

   After the pull, we must re-configure Gnuastro with ‘autoreconf -f’
(part of GNU Autoconf).  It will update the ‘./configure’ script and all
the ‘Makefile.in’(2) files based on the hand-written configurations (in
‘configure.ac’ and the ‘Makefile.am’ files).  After running ‘autoreconf
-f’, a warning about ‘TEXI2DVI’ might show up, you can ignore that.

   The most important reason for re-building Gnuastro’s build system is
to generate/update the version number for your updated Gnuastro
snapshot.  This generated version number will include the commit
information (see *note Version numbering::).  The version number is
included in nearly all outputs of Gnuastro’s programs, therefore it is
vital for reproducing an old result.

   As a summary, be sure to run ‘‘autoreconf -f’’ after every change in
the Git history.  This includes synchronization with the main server or
even a commit you have made yourself.

   If you would like to see what has changed since you last synchronized
your local clone, you can take the following steps instead of the simple
command above (do not type anything after ‘#’):

     $ git checkout master             # Confirm if you are on master.
     $ git fetch origin                # Fetch all new commits from server.
     $ git log master..origin/master   # See all the new commit messages.
     $ git merge origin/master         # Update your master branch.
     $ autoreconf -f                   # Update the build system.

By default ‘git log’ prints the most recent commit first, add the
‘--reverse’ option to see the changes chronologically.  To see exactly
what has been changed in the source code along with the commit message,
add a ‘-p’ option to the ‘git log’.

   If you want to make changes in the code, have a look at *note
Developing:: to get started easily.  Be sure to commit your changes in a
separate branch (keep your ‘master’ branch to follow the official
repository) and re-run ‘autoreconf -f’ after the commit.  If you intend
to send your work to us, you can safely use your commit since it will be
ultimately recorded in Gnuastro’s official history.  If not, please
upload your separate branch to a public hosting service, for example,
Codeberg (https://codeberg.org), and link to it in your report/paper.
Alternatively, run ‘make distcheck’ and upload the output
‘gnuastro-X.X.X.XXXX.tar.gz’ to a publicly accessible web page so your
results can be considered scientific (reproducible) later.

   ---------- Footnotes ----------

   (1) <https://savannah.gnu.org/task/index.php?13993> is defined for
you to check if significant (for Gnuastro) updates are made in these
repositories, since the last time you pulled from them.

   (2) In the GNU build system, ‘./configure’ will use the ‘Makefile.in’
files to create the necessary ‘Makefile’ files that are later read by
‘make’ to build the package.


File: gnuastro.info,  Node: Build and install,  Prev: Downloading the source,  Up: Installation

3.3 Build and install
=====================

This section is basically a longer explanation to the sequence of
commands given in *note Quick start::.  If you did not have any problems
during the *note Quick start:: steps, you want to have all the programs
of Gnuastro installed in your system, you do not want to change the
executable names during or after installation, you have root access to
install the programs in the default system wide directory, the Letter
paper size of the print book is fine for you or as a summary you do not
feel like going into the details when everything is working, you can
safely skip this section.

   If you have any of the above problems or you want to understand the
details for a better control over your build and install, read along.
The dependencies which you will need prior to configuring, building and
installing Gnuastro are explained in *note Dependencies::.  The first
three steps in *note Quick start:: need no extra explanation, so we will
skip them and start with an explanation of Gnuastro specific
configuration options and a discussion on the installation directory in
*note Configuring::, followed by some smaller subsections: *note
Tests::, *note A4 print book::, and *note Known issues:: which explains
the solutions to known problems you might encounter in the installation
steps and ways you can solve them.

* Menu:

* Configuring::                 Configure Gnuastro
* Separate build and source directories::  Keeping derivate/build files separate.
* Tests::                       Run tests to see if it is working.
* A4 print book::               Customize the print book.
* Known issues::                Issues you might encounter.


File: gnuastro.info,  Node: Configuring,  Next: Separate build and source directories,  Prev: Build and install,  Up: Build and install

3.3.1 Configuring
-----------------

The ‘$ ./configure’ step is the most important step in the build and
install process.  All the required packages, libraries, headers and
environment variables are checked in this step.  The behaviors of make
and make install can also be set through command-line options to this
command.

   The configure script accepts various arguments and options which
enable the final user to highly customize whatever she is building.  The
options to configure are generally very similar to normal program
options explained in *note Arguments and options::.  Similar to all GNU
programs, you can get a full list of the options along with a short
explanation by running

     $ ./configure --help

A complete explanation is also included in the ‘INSTALL’ file.  Note
that this file was written by the authors of GNU Autoconf (which builds
the ‘configure’ script), therefore it is common for all programs which
use the ‘$ ./configure’ script for building and installing, not just
Gnuastro.  Here we only discuss cases where you do not have superuser
access to the system and if you want to change the executable names.
But before that, a review of the options to configure that are
particular to Gnuastro are discussed.

* Menu:

* Gnuastro configure options::  Configure options particular to Gnuastro.
* Installation directory::      Specify the directory to install.
* Executable names::            Changing executable names.
* Configure and build in RAM::  For minimal use of HDD or SSD, and clean source.


File: gnuastro.info,  Node: Gnuastro configure options,  Next: Installation directory,  Prev: Configuring,  Up: Configuring

3.3.1.1 Gnuastro configure options
..................................

Most of the options to configure (which are to do with building) are
similar for every program which uses this script.  Here the options that
are particular to Gnuastro are discussed.  The next topics explain the
usage of other configure options which can be applied to any program
using the GNU build system (through the configure script).

‘--enable-debug’
     Compile/build Gnuastro with debugging information, no optimization
     and without shared libraries.

     In order to allow more efficient programs when using Gnuastro
     (after the installation), by default Gnuastro is built with a 3rd
     level (a very high level) optimization and no debugging
     information.  By default, libraries are also built for static _and_
     shared linking (see *note Linking::).  However, when there are
     crashes or unexpected behavior, these three features can hinder the
     process of localizing the problem.  This configuration option is
     identical to manually calling the configuration script with
     ‘CFLAGS="-g -O0" --disable-shared’.

     In the (rare) situations where you need to do your debugging on the
     shared libraries, do not use this option.  Instead run the
     configure script by explicitly setting ‘CFLAGS’ like this:
          $ ./configure CFLAGS="-g -O0"

‘--enable-check-with-valgrind’
     Do the ‘make check’ tests through Valgrind.  Therefore, if any
     crashes or memory-related issues (segmentation faults in
     particular) occur in the tests, the output of Valgrind will also be
     put in the ‘tests/test-suite.log’ file without having to manually
     modify the check scripts.  This option will also activate
     Gnuastro’s debug mode (see the ‘--enable-debug’ configure-time
     option described above).

     Valgrind is free software.  It is a program for easy checking of
     memory-related issues in programs.  It runs a program within its
     own controlled environment and can thus identify the exact
     line-number in the program’s source where a memory-related issue
     occurs.  However, it can significantly slow-down the tests.  So
     this option is only useful when a segmentation fault is found
     during ‘make check’.

‘--enable-progname’
     Only build and install ‘progname’ along with any other program that
     is enabled in this fashion.  ‘progname’ is the name of the
     executable without the ‘ast’, for example, ‘crop’ for Crop (with
     the executable name of ‘astcrop’).

     Note that by default all the programs will be installed.  This
     option (and the ‘--disable-progname’ options) are only relevant
     when you do not want to install all the programs.  Therefore, if
     this option is called for any of the programs in Gnuastro, any
     program which is not explicitly enabled will not be built or
     installed.

‘--disable-progname’
‘--enable-progname=no’
     Do not build or install the program named ‘progname’.  This is very
     similar to the ‘--enable-progname’, but will build and install all
     the other programs except this one.

     *Note:* If some programs are enabled and some are disabled, it is
     equivalent to simply enabling those that were enabled.  Listing the
     disabled programs is redundant.

‘--enable-gnulibcheck’
     Enable checks on the GNU Portability Library (Gnulib).  Gnulib is
     used by Gnuastro to enable users of non-GNU based operating systems
     (that do not use GNU C library or glibc) to compile and use the
     advanced features that this library provides.  We make extensive
     use of such functions.  If you give this option to ‘$ ./configure’,
     when you run ‘$ make check’, first the functions in Gnulib will be
     tested, then the Gnuastro executables.  If your operating system
     does not support glibc or has an older version of it and you have
     problems in the build process (‘$ make’), you can give this flag to
     configure to see if the problem is caused by Gnulib not supporting
     your operating system or Gnuastro, see *note Known issues::.

‘--disable-guide-message’
‘--enable-guide-message=no’
     Do not print a guiding message during the GNU Build process of
     *note Quick start::.  By default, after each step, a message is
     printed guiding the user what the next command should be.
     Therefore, after ‘./configure’, it will suggest running ‘make’.
     After ‘make’, it will suggest running ‘make check’ and so on.  If
     Gnuastro is configured with this option, for example
          $ ./configure --disable-guide-message
     Then these messages will not be printed after any step (like most
     programs).  For people who are not yet fully accustomed to this
     build system, these guidelines can be very useful and encouraging.
     However, if you find those messages annoying, use this option.

‘--without-libgit2’
     Build Gnuastro without libgit2 (for including Git commit hashes in
     output files), see *note Optional dependencies::.  libgit2 is an
     optional dependency, with this option, Gnuastro will ignore any
     possibly existing libgit2 that may already be on the system.

‘--without-libjpeg’
     Build Gnuastro without libjpeg (for reading/writing to JPEG files),
     see *note Optional dependencies::.  libjpeg is an optional
     dependency, with this option, Gnuastro will ignore any possibly
     existing libjpeg that may already be on the system.

‘--without-libtiff’
     Build Gnuastro without libtiff (for reading/writing to TIFF files),
     see *note Optional dependencies::.  libtiff is an optional
     dependency, with this option, Gnuastro will ignore any possibly
     existing libtiff that may already be on the system.

‘--with-python’
     Build the Python interface within Gnuastro’s dynamic library.  This
     interface can be used for easy communication with Python wrappers
     (for example, the pyGnuastro package).

     When you install the pyGnuastro package from PyPI, the correct
     configuration of the Gnuastro Library is already packaged with it
     (with the Python interface) and that is independent of your
     Gnuastro installation.  The Python interface is only necessary if
     you want to build pyGnuastro from source (which is only necessary
     for developers).  Therefore it has to be explicitly activated at
     configure time with this option.  For more on the interface
     functions, see *note Python interface::.

   The tests of some programs might depend on the outputs of the tests
of other programs.  For example, MakeProfiles is one the first programs
to be tested when you run ‘$ make check’.  MakeProfiles’ test outputs
(FITS images) are inputs to many other programs (which in turn provide
inputs for other programs).  Therefore, if you do not install
MakeProfiles for example, the tests for many the other programs will be
skipped.  To avoid this, in one run, you can install all the programs
and run the tests but not install.  If everything is working correctly,
you can run configure again with only the programs you want.  However,
do not run the tests and directly install after building.


File: gnuastro.info,  Node: Installation directory,  Next: Executable names,  Prev: Gnuastro configure options,  Up: Configuring

3.3.1.2 Installation directory
..............................

One of the most commonly used options to ‘./configure’ is ‘--prefix’, it
is used to define the directory that will host all the installed files
(or the “prefix” in their final absolute file name).  For example, when
you are using a server and you do not have administrator or root access.
In this example scenario, if you do not use the ‘--prefix’ option, you
will not be able to install the built files and thus access them from
anywhere without having to worry about where they are installed.
However, once you prepare your startup file to look into the proper
place (as discussed thoroughly below), you will be able to easily use
this option and benefit from any software you want to install without
having to ask the system administrators or install and use a different
version of a software that is already installed on the server.

   The most basic way to run an executable is to explicitly write its
full file name (including all the directory information) and run it.
One example is running the configuration script with the ‘$ ./configure’
command (see *note Quick start::).  By giving a specific directory (the
current directory or ‘./’), we are explicitly telling the shell to look
in the current directory for an executable file named ‘‘configure’’.
Directly specifying the directory is thus useful for executables in the
current (or nearby) directories.  However, when the program (an
executable file) is to be used a lot, specifying all those directories
will become a significant burden.  For example, the ‘ls’ executable
lists the contents in a given directory and it is (usually) installed in
the ‘/usr/bin/’ directory by the operating system maintainers.
Therefore, if using the full address was the only way to access an
executable, each time you wanted a listing of a directory, you would
have to run the following command (which is very inconvenient, both in
writing and in remembering the various directories).

     $ /usr/bin/ls

   To address this problem, we have the ‘PATH’ environment variable.  To
understand it better, we will start with a short introduction to the
shell variables.  Shell variable values are basically treated as strings
of characters.  For example, it does not matter if the value is a name
(string of _alphabetic_ characters), or a number (string of _numeric_
characters), or both.  You can define a variable and a value for it by
running
     $ myvariable1=a_test_value
     $ myvariable2="a test value"
As you see above, if the value contains white space characters, you have
to put the whole value (including white space characters) in double
quotes (<">).  You can see the value it represents by running
     $ echo $myvariable1
     $ echo $myvariable2
If a variable has no value or it was not defined, the last command will
only print an empty line.  A variable defined like this will be known as
long as this shell or terminal is running.  Other terminals will have no
idea it existed.  The main advantage of shell variables is that if they
are exported(1), subsequent programs that are run within that shell can
access their value.  So by changing their value, you can change the
“environment” of a program which uses them.  The shell variables which
are accessed by programs are therefore known as “environment
variables”(2).  You can see the full list of exported variables that
your shell recognizes by running:

     $ printenv

   ‘HOME’ is one commonly used environment variable, it is any user’s
(the one that is logged in) top directory.  Try finding it in the
command above.  It is used so often that the shell has a special
expansion (alternative) for it: ‘‘~’’.  Whenever you see file names
starting with the tilde sign, it actually represents the value to the
‘HOME’ environment variable, so ‘~/doc’ is the same as ‘$HOME/doc’.

   Another one of the most commonly used environment variables is
‘PATH’, it is a list of directories to search for executable names.  Its
value is a list of directories (separated by a colon, or ‘<:>’).  When
the address of the executable is not explicitly given (like
‘./configure’ above), the system will look for the executable in the
directories specified by ‘PATH’.  If you have a computer nearby, try
running the following command to see which directories your system will
look into when it is searching for executable (binary) files, one
example is printed here (notice how ‘/usr/bin’, in the ‘ls’ example
above, is one of the directories in ‘PATH’):

     $ echo $PATH
     /usr/local/sbin:/usr/local/bin:/usr/bin

   By default ‘PATH’ usually contains system-wide directories, which are
readable (but not writable) by all users, like the above example.
Therefore if you do not have root (or administrator) access, you need to
add another directory to ‘PATH’ which you actually have write access to.
The standard directory where you can keep installed files (not just
executables) for your own user is the ‘~/.local/’ directory.  The names
of hidden files start with a ‘<.>’ (dot), so it will not show up in your
common command-line listings, or on the graphical user interface.  You
can use any other directory, but this is the most recognized.

   The top installation directory will be used to keep all the package’s
components: programs (executables), libraries, include (header) files,
shared data (like manuals), or configuration files (see *note Review of
library fundamentals:: for a thorough introduction to headers and
linking).  So it commonly has some of the following sub-directories for
each class of installed components respectively: ‘bin/’, ‘lib/’,
‘include/’ ‘man/’, ‘share/’, ‘etc/’.  Since the ‘PATH’ variable is only
used for executables, you can add the ‘~/.local/bin’ directory (which
keeps the executables/programs or more generally, “binary” files) to
‘PATH’ with the following command.  As defined below, first the existing
value of ‘PATH’ is used, then your given directory is added to its end
and the combined value is put back in ‘PATH’ (run ‘‘$ echo $PATH’’
afterwards to check if it was added).

     $ PATH=$PATH:~/.local/bin

   Any executable that you installed in ‘~/.local/bin’ will now be
usable without having to remember and write its full address.  However,
as soon as you leave/close your current terminal session, this modified
‘PATH’ variable will be forgotten.  Adding the directories which contain
executables to the ‘PATH’ environment variable each time you start a
terminal is also very inconvenient and prone to errors.  Fortunately,
there are standard ‘startup files’ defined by your shell precisely for
this (and other) purposes.  There is a special startup file for every
significant starting step:

‘/etc/profile’ and everything in ‘/etc/profile.d/’
     These startup scripts are called when your whole system starts (for
     example, after you turn on your computer).  Therefore you need
     administrator or root privileges to access or modify them.

‘~/.bash_profile’
     If you are using (GNU) Bash as your shell, the commands in this
     file are run, when you log in to your account _through Bash_.  Most
     commonly when you login through the virtual console (where there is
     no graphic user interface).

‘~/.bashrc’
     If you are using (GNU) Bash as your shell, the commands here will
     be run each time you start a terminal and are already logged in.
     For example, when you open your terminal emulator in the graphic
     user interface.

   For security reasons, it is highly recommended to directly type in
your ‘HOME’ directory value by hand in startup files instead of using
variables.  So in the following, let’s assume your user name is ‘‘name’’
(so ‘~’ may be replaced with ‘/home/name’).  To add ‘~/.local/bin’ to
your ‘PATH’ automatically on any startup file, you have to “export” the
new value of ‘PATH’ in the startup file that is most relevant to you by
adding this line:

     export PATH=$PATH:/home/name/.local/bin

   Now that you know your system will look into ‘~/.local/bin’ for
executables, you can tell Gnuastro’s configure script to install
everything in the top ‘~/.local’ directory using the ‘--prefix’ option.
When you subsequently run ‘$ make install’, all the install-able files
will be put in their respective directory under ‘~/.local/’ (the
executables in ‘~/.local/bin’, the compiled library files in
‘~/.local/lib’, the library header files in ‘~/.local/include’ and so
on, to learn more about these different files, please see *note Review
of library fundamentals::).  Note that tilde (‘<~>’) expansion will not
happen if you put a ‘<=>’ between ‘--prefix’ and ‘~/.local’(3), so we
have avoided the <=> character here which is optional in GNU-style
options, see *note Options::.

     $ ./configure --prefix ~/.local

   You can install everything (including libraries like GSL, CFITSIO, or
WCSLIB which are Gnuastro’s mandatory dependencies, see *note Mandatory
dependencies::) locally by configuring them as above.  However, recall
that ‘PATH’ is only for executable files, not libraries and that
libraries can also depend on other libraries.  For example, WCSLIB
depends on CFITSIO and Gnuastro needs both.  Therefore, when you
installed a library in a non-recognized directory, you have to guide the
program that depends on them to look into the necessary library and
header file directories.  To do that, you have to define the ‘LDFLAGS’
and ‘CPPFLAGS’ environment variables respectively.  This can be done
while calling ‘./configure’ as shown below:

     $ ./configure LDFLAGS=-L/home/name/.local/lib            \
                   CPPFLAGS=-I/home/name/.local/include       \
                   --prefix ~/.local

   It can be annoying/buggy to do this when configuring every software
that depends on such libraries.  Hence, you can define these two
variables in the most relevant startup file (discussed above).  The
convention on using these variables does not include a colon to separate
values (as ‘PATH’-like variables do).  They use white space characters
and each value is prefixed with a compiler option(4).  Note the ‘-L’ and
‘-I’ above (see *note Options::), for ‘-I’ see *note Headers::, and for
‘-L’, see *note Linking::.  Therefore we have to keep the value in
double quotation signs to keep the white space characters and adding the
following two lines to the startup file of choice:

     export LDFLAGS="$LDFLAGS -L/home/name/.local/lib"
     export CPPFLAGS="$CPPFLAGS -I/home/name/.local/include"

   Dynamic libraries are linked to the executable every time you run a
program that depends on them (see *note Linking:: to fully understand
this important concept).  Hence dynamic libraries also require a special
path variable called ‘LD_LIBRARY_PATH’ (same formatting as ‘PATH’).  To
use programs that depend on these libraries, you need to add
‘~/.local/lib’ to your ‘LD_LIBRARY_PATH’ environment variable by adding
the following line to the relevant start-up file:

     export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/name/.local/lib

   If you also want to access the Info (see *note Info::) and man pages
(see *note Man pages::) documentations add ‘~/.local/share/info’ and
‘~/.local/share/man’ to your ‘INFOPATH’(5) and ‘MANPATH’ environment
variables respectively.

   A final note is that order matters in the directories that are
searched for all the variables discussed above.  In the examples above,
the new directory was added after the system specified directories.  So
if the program, library or manuals are found in the system wide
directories, the user directory is no longer searched.  If you want to
search your local installation first, put the new directory before the
already existing list, like the example below.

     export LD_LIBRARY_PATH=/home/name/.local/lib:$LD_LIBRARY_PATH

This is good when a library, for example, CFITSIO, is already present on
the system, but the system-wide install was not configured with the
correct configuration flags (see *note CFITSIO::), or you want to use a
newer version and you do not have administrator or root access to update
it on the whole system/server.  If you update ‘LD_LIBRARY_PATH’ by
placing ‘~/.local/lib’ first (like above), the linker will first find
the CFITSIO you installed for yourself and link with it.  It thus will
never reach the system-wide installation.

   There are important security problems with using local installations
first: all important system-wide executables and libraries (important
executables like ‘ls’ and ‘cp’, or libraries like the C library) can be
replaced by non-secure versions with the same file names and put in the
customized directory (‘~/.local’ in this example).  So if you choose to
search in your customized directory first, please _be sure_ to keep it
clean from executables or libraries with the same names as important
system programs or libraries.

*Summary:* When you are using a server which does not give you
administrator/root access AND you would like to give priority to your
own built programs and libraries, not the version that is (possibly
already) present on the server, add these lines to your startup file.
See above for which startup file is best for your case and for a
detailed explanation on each.  Do Not forget to replace
‘‘/YOUR-HOME-DIR’’ with your home directory (for example,
‘‘/home/your-id’’):

     export PATH="/YOUR-HOME-DIR/.local/bin:$PATH"
     export LDFLAGS="-L/YOUR-HOME-DIR/.local/lib $LDFLAGS"
     export MANPATH="/YOUR-HOME-DIR/.local/share/man/:$MANPATH"
     export CPPFLAGS="-I/YOUR-HOME-DIR/.local/include $CPPFLAGS"
     export INFOPATH="/YOUR-HOME-DIR/.local/share/info/:$INFOPATH"
     export LD_LIBRARY_PATH="/YOUR-HOME-DIR/.local/lib:$LD_LIBRARY_PATH"

Afterwards, you just need to add an extra
‘--prefix=/YOUR-HOME-DIR/.local’ to the ‘./configure’ command of the
software that you intend to install.  Everything else will be the same
as a standard build and install, see *note Quick start::.

   ---------- Footnotes ----------

   (1) By running ‘$ export myvariable=a_test_value’ instead of the
simpler case in the text

   (2) You can use shell variables for other actions too, for example,
to temporarily keep some names or run loops on some files.

   (3) If you insist on using ‘<=>’, you can use
‘--prefix=$HOME/.local’.

   (4) These variables are ultimately used as options while building the
programs.  Therefore every value has be an option name followed be a
value as discussed in *note Options::.

   (5) Info has the following convention: “If the value of ‘INFOPATH’
ends with a colon [or it is not defined] ..., the initial list of
directories is constructed by appending the build-time default to the
value of ‘INFOPATH’.” So when installing in a non-standard directory and
if ‘INFOPATH’ was not initially defined, add a colon to the end of
‘INFOPATH’ as shown below.  Otherwise Info will not be able to find
system-wide installed documentation:
‘echo 'export INFOPATH=$INFOPATH:/home/name/.local/share/info:' >>
~/.bashrc’
Note that this is only an internal convention of Info: do not use it for
other ‘*PATH’ variables.


File: gnuastro.info,  Node: Executable names,  Next: Configure and build in RAM,  Prev: Installation directory,  Up: Configuring

3.3.1.3 Executable names
........................

At first sight, the names of the executables for each program might seem
to be uncommonly long, for example, ‘astnoisechisel’ or ‘astcrop’.  We
could have chosen terse (and cryptic) names like most programs do.  We
chose this complete naming convention (something like the commands in
TeX) so you do not have to spend too much time remembering what the name
of a specific program was.  Such complete names also enable you to
easily search for the programs.

   To facilitate typing the names in, we suggest using the shell
auto-complete.  With this facility you can find the executable you want
very easily.  It is very similar to file name completion in the shell.
For example, simply by typing the letters below (where <[TAB]> stands
for the Tab key on your keyboard)

     $ ast[TAB][TAB]

you will get the list of all the available executables that start with
‘ast’ in your ‘PATH’ environment variable directories.  So, all the
Gnuastro executables installed on your system will be listed.  Typing
the next letter for the specific program you want along with a Tab, will
limit this list until you get to your desired program.

   In case all of this does not convince you and you still want to type
short names, some suggestions are given below.  You should have in mind
though, that if you are writing a shell script that you might want to
pass on to others, it is best to use the standard name because other
users might not have adopted the same customization.  The long names
also serve as a form of documentation in such scripts.  A similar
reasoning can be given for option names in scripts: it is good practice
to always use the long formats of the options in shell scripts, see
*note Options::.

   The simplest solution is making a symbolic link to the actual
executable.  For example, let’s assume you want to type ‘ic’ to run Crop
instead of ‘astcrop’.  Assuming you installed Gnuastro executables in
‘/usr/local/bin’ (default) you can do this simply by running the
following command as root:

     # ln -s /usr/local/bin/astcrop /usr/local/bin/ic

In case you update Gnuastro and a new version of Crop is installed, the
default executable name is the same, so your custom symbolic link still
works.

   The installed executable names can also be set using options to ‘$
./configure’, see *note Configuring::.  GNU Autoconf (which configures
Gnuastro for your particular system), allows the builder to change the
name of programs with the three options ‘--program-prefix’,
‘--program-suffix’ and ‘--program-transform-name’.  The first two are
for adding a fixed prefix or suffix to all the programs that will be
installed.  This will actually make all the names longer!  You can use
it to add versions of program names to the programs in order to
simultaneously have two executable versions of a program.

   The third configure option allows you to set the executable name at
install time using the SED program.  SED is a very useful ‘stream
editor’.  There are various resources on the internet to use it
effectively.  However, we should caution that using configure options
will change the actual executable name of the installed program and on
every re-install (an update for example), you have to also add this
option to keep the old executable name updated.  Also note that the
documentation or configuration files do not change from their standard
names either.

   For example, let’s assume that typing ‘ast’ on every invocation of
every program is really annoying you!  You can remove this prefix from
all the executables at configure time by adding this option:

     $ ./configure --program-transform-name='s/ast/ /'


File: gnuastro.info,  Node: Configure and build in RAM,  Prev: Executable names,  Up: Configuring

3.3.1.4 Configure and build in RAM
..................................

Gnuastro’s configure and build process (the GNU build system) involves
the creation, reading, and modification of a large number of files
(input/output, or I/O). Therefore file I/O issues can directly affect
the work of developers who need to configure and build Gnuastro numerous
times.  Some of these issues are listed below:

   • I/O will cause wear and tear on both the HDDs (mechanical failures)
     and SSDs (decreasing the lifetime).

   • Having the built files mixed with the source files can greatly
     affect backing up (synchronization) of source files (since it
     involves the management of a large number of small files that are
     regularly changed.  Backup software can of course be configured to
     ignore the built files and directories.  However, since the built
     files are mixed with the source files and can have a large variety,
     this will require a high level of customization.

   One solution to address both these problems is to use the tmpfs file
system (https://en.wikipedia.org/wiki/Tmpfs).  Any file in tmpfs is
actually stored in the RAM (and possibly SWAP), not on HDDs or SSDs.
The RAM is built for extensive and fast I/O. Therefore the large number
of file I/Os associated with configuring and building will not harm the
HDDs or SSDs.  Due to the volatile nature of RAM, files in the tmpfs
file-system will be permanently lost after a power-off.  Since all
configured and built files are derivative files (not files that have
been directly written by hand) there is no problem in this and this
feature can be considered as an automatic cleanup.

   The modern GNU C library (and thus the Linux kernel) defines the
‘/dev/shm’ directory for this purpose in the RAM (POSIX shared memory).
To build in it, you can use the GNU build system’s ability to build in a
separate directory (not necessarily in the source directory) as shown
below.  Just set ‘SRCDIR’ as the address of Gnuastro’s top source
directory (for example, where there is the unpacked tarball).

     $ SRCDIR=/home/username/gnuastro
     $ mkdir /dev/shm/tmp-gnuastro-build
     $ cd /dev/shm/tmp-gnuastro-build
     $ $SRCDIR/configure --srcdir=$SRCDIR
     $ make

   Gnuastro comes with a script to simplify this process of configuring
and building in a different directory (a “clean” build), for more see
*note Separate build and source directories::.


File: gnuastro.info,  Node: Separate build and source directories,  Next: Tests,  Prev: Configuring,  Up: Build and install

3.3.2 Separate build and source directories
-------------------------------------------

The simple steps of *note Quick start:: will mix the source and built
files.  This can cause inconvenience for developers or enthusiasts
following the most recent work (see *note Version controlled source::).
The current section is mainly focused on this later group of Gnuastro
users.  If you just install Gnuastro on major releases (following *note
Announcements::), you can safely ignore this section.

   When it is necessary to keep the source (which is under version
control), but not the derivative (built) files (after checking or
installing), the best solution is to keep the source and the built files
in separate directories.  One application of this is already discussed
in *note Configure and build in RAM::.

   To facilitate this process of configuring and building in a separate
directory, Gnuastro comes with the ‘developer-build’ script.  It is
available in the top source directory and is _not_ installed.  It will
make a directory under a given top-level directory (given to
‘--top-build-dir’) and build Gnuastro there.  It thus keeps the source
completely separated from the built files.  For easy access to the built
files, it also makes a symbolic link to the built directory in the top
source files called ‘build’.

   When running the developer-build script without any options in the
Gnuastro’s top source directory, default values will be used for its
configuration.  As with Gnuastro’s programs, you can inspect the default
values with ‘-P’ (or ‘--printparams’, the output just looks a little
different here).  The default top-level build directory is ‘/dev/shm’:
the shared memory directory in RAM on GNU/Linux systems as described in
*note Configure and build in RAM::.

   Besides these, it also has some features to facilitate the job of
developers or bleeding edge users like the ‘--debug’ option to do a fast
build, with debug information, no optimization, and no shared libraries.
Here is the full list of options you can feed to this script to
configure its operations.

*Not all Gnuastro’s common program behavior usable here:*
‘developer-build’ is just a non-installed script with a very limited
scope as described above.  It thus does not have all the common option
behaviors or configuration files for example.

*White space between option and value:* ‘developer-build’ does not
accept an <=> sign between the options and their values.  It also needs
at least one character between the option and its value.  Therefore ‘-n
4’ or ‘--numthreads 4’ are acceptable, while ‘-n4’, ‘-n=4’, or
‘--numthreads=4’ are not.  Finally multiple short option names cannot be
merged: for example, you can say ‘-c -n 4’, but unlike Gnuastro’s
programs, ‘-cn4’ is not acceptable.

*Reusable for other packages:* This script can be used in any software
which is configured and built using the GNU Build System.  Just copy it
in the top source directory of that software and run it from there.

*Example usage:* See *note Forking tutorial:: for an example usage of
this script in some scenarios.

‘-b STR’
‘--top-build-dir STR’
     The top build directory to make a directory for the build.  If this
     option is not called, the top build directory is ‘/dev/shm’ (only
     available in GNU/Linux operating systems, see *note Configure and
     build in RAM::).

‘-V’
‘--version’
     Print the version string of Gnuastro that will be used in the
     build.  This string will be appended to the directory name
     containing the built files.

‘-a’
‘--autoreconf’
     Run ‘autoreconf -f’ before building the package.  In Gnuastro, this
     is necessary when a new commit has been made to the project
     history.  In Gnuastro’s build system, the Git description will be
     used as the version, see *note Version numbering:: and *note
     Synchronizing::.

‘-c’
‘--clean’
     Delete the contents of the build directory (clean it) before
     starting the configuration and building of this run.

     This is useful when you have recently pulled changes from the main
     Git repository, or committed a change yourself and ran ‘autoreconf
     -f’, see *note Synchronizing::.  After running GNU Autoconf, the
     version will be updated and you need to do a clean build.

‘-d’
‘--debug’
     Build with debugging flags (for example, to use in GNU Debugger,
     also known as GDB, or Valgrind), disable optimization and also the
     building of shared libraries.  Similar to running the configure
     script of below

          $ ./configure --enable-debug

     Besides all the debugging advantages of building with this option,
     it will also be significantly speed up the build (at the cost of
     slower built programs).  So when you are testing something small or
     working on the build system itself, it will be much faster to test
     your work with this option.

‘-v’
‘--valgrind’
     Build all ‘make check’ tests within Valgrind.  For more, see the
     description of ‘--enable-check-with-valgrind’ in *note Gnuastro
     configure options::.

‘-j INT’
‘--jobs INT’
     The maximum number of threads/jobs for Make to build at any moment.
     As the name suggests (Make has an identical option), the number
     given to this option is directly passed on to any call of Make with
     its ‘-j’ option.

‘-C’
‘--check’
     After finishing the build, also run ‘make check’.  By default,
     ‘make check’ is not run because the developer usually has their own
     checks to work on (for example, defined in ‘tests/during-dev.sh’).

‘-i’
‘--install’
     After finishing the build, also run ‘make install’.

‘-D’
‘--dist’
     Run ‘make dist-lzip pdf’ to build a distribution tarball (in
     ‘.tar.lz’ format) and a PDF manual.  This can be useful for
     archiving, or sending to colleagues who do not use Git for an easy
     build and manual.

‘-u STR’
‘--upload STR’
     Activate the ‘--dist’ (‘-D’) option, then use secure copy (‘scp’,
     part of the SSH tools) to copy the tarball and PDF to the ‘src’ and
     ‘pdf’ sub-directories of the specified server and its directory
     (value to this option).  For example, ‘--upload my-server:dir’,
     will copy the tarball in the ‘dir/src’, and the PDF manual in
     ‘dir/pdf’ of ‘my-server’ server.  It will then make a symbolic link
     in the top server directory to the tarball that is called
     ‘gnuastro-latest.tar.lz’.

‘-p STR’
‘--publish=STR’
     Clean, bootstrap, build, check and upload the checked tarball and
     PDF of the book to the URL given as ‘STR’.  This option is just a
     wrapper for ‘--autoreconf --clean --debug --check --upload STR’.
     ‘--debug’ is added because it will greatly speed up the build.
     ‘--debug’ will have no effect on the produced tarball (people who
     later download will be building with the default optimized, and
     non-debug mode).  This option is good when you have made a commit
     and are ready to publish it on your server (if nothing crashes).
     Recall that if any of the previous steps fail the script aborts.

‘-I’
‘--install-archive’
     Short for ‘--autoreconf --clean --check --install --dist’.  This is
     useful when you actually want to install the commit you just made
     (if the build and checks succeed).  It will also produce a
     distribution tarball and PDF manual for easy access to the
     installed tarball on your system at a later time.

     Ideally, Gnuastro’s Git version history makes it easy for a
     prepared system to revert back to a different point in history.
     But Gnuastro also needs to bootstrap files and also your
     collaborators might (usually do!)  find it too much of a burden to
     do the bootstrapping themselves.  So it is convenient to have a
     tarball and PDF manual of the version you have installed (and are
     using in your research) handily available.

‘-h’
‘--help’
‘-P’
‘--printparams’
     Print a description of this script along with all the options and
     their current values.


File: gnuastro.info,  Node: Tests,  Next: A4 print book,  Prev: Separate build and source directories,  Up: Build and install

3.3.3 Tests
-----------

After successfully building (compiling) the programs with the ‘$ make’
command you can check the installation before installing.  To run the
tests, run

     $ make check

   For every program some tests are designed to check some possible
operations.  Running the command above will run those tests and give you
a final report.  If everything is OK and you have built all the
programs, all the tests should pass.  In case any of the tests fail,
please have a look at *note Known issues:: and if that still does not
fix your problem, look that the ‘./tests/test-suite.log’ file to see if
the source of the error is something particular to your system or more
general.  If you feel it is general, please contact us because it might
be a bug.  Note that the tests of some programs depend on the outputs of
other program’s tests, so if you have not installed them they might be
skipped or fail.  Prior to releasing every distribution all these tests
are checked.  If you have a reasonably modern terminal, the outputs of
the successful tests will be colored green and the failed ones will be
colored red.

   These scripts can also act as a good set of examples for you to see
how the programs are run.  All the tests are in the ‘tests/’ directory.
The tests for each program are shell scripts (ending with ‘.sh’) in a
sub-directory of this directory with the same name as the program.  See
*note Test scripts:: for more detailed information about these scripts
in case you want to inspect them.


File: gnuastro.info,  Node: A4 print book,  Next: Known issues,  Prev: Tests,  Up: Build and install

3.3.4 A4 print book
-------------------

The default print version of this book is provided in the letter paper
size.  If you would like to have the print version of this book on paper
and you are living in a country which uses A4, then you can rebuild the
book.  The great thing about the GNU build system is that the book
source code which is in Texinfo is also distributed with the program
source code, enabling you to do such customization (hacking).

   In order to change the paper size, you will need to have GNU Texinfo
installed.  Open ‘doc/gnuastro.texi’ with any text editor.  This is the
source file that created this book.  In the first few lines you will see
this line:

     @c@afourpaper

In Texinfo, a line is commented with ‘@c’.  Therefore, un-comment this
line by deleting the first two characters such that it changes to:

     @afourpaper

Save the file and close it.  You can now run the following command

     $ make pdf

and the new PDF book will be available in ‘SRCdir/doc/gnuastro.pdf’.  By
changing the ‘pdf’ in ‘$ make pdf’ to ‘ps’ or ‘dvi’ you can have the
book in those formats.  Note that you can do this for any book that is
in Texinfo format, they might not have ‘@afourpaper’ line, so you can
add it close to the top of the Texinfo source file.


File: gnuastro.info,  Node: Known issues,  Prev: A4 print book,  Up: Build and install

3.3.5 Known issues
------------------

Depending on your operating system and the version of the compiler you
are using, you might confront some known problems during the
configuration (‘$ ./configure’), compilation (‘$ make’) and tests (‘$
make check’).  Here, their solutions are discussed.

   • ‘$ ./configure’: _Configure complains about not finding a library
     even though you have installed it._  The possible solution is based
     on how you installed the package:

        • From your distribution’s package manager.  Most probably this
          is because your distribution has separated the header files of
          a library from the library parts.  Please also install the
          ‘development’ packages for those libraries too.  Just add a
          ‘-dev’ or ‘-devel’ to the end of the package name and re-run
          the package manager.  This will not happen if you install the
          libraries from source.  When installed from source, the
          headers are also installed.

        • From source.  Then your linker is not looking where you
          installed the library.  If you followed the instructions in
          this chapter, all the libraries will be installed in
          ‘/usr/local/lib’.  So you have to tell your linker to look in
          this directory.  To do so, configure Gnuastro like this:

               $ ./configure LDFLAGS="-L/usr/local/lib"

          If you want to use the libraries for your other programming
          projects, then export this environment variable in a start-up
          script similar to the case for ‘LD_LIBRARY_PATH’ explained
          below, also see *note Installation directory::.

   • ‘$ make’: _Complains about an unknown function on a non-GNU based
     operating system._  In this case, please run ‘$ ./configure’ with
     the ‘--enable-gnulibcheck’ option to see if the problem is from the
     GNU Portability Library (Gnulib) not supporting your system or if
     there is a problem in Gnuastro, see *note Gnuastro configure
     options::.  If the problem is not in Gnulib and after all its tests
     you get the same complaint from ‘make’, then please contact us at
     ‘bug-gnuastro@gnu.org’.  The cause is probably that a function that
     we have used is not supported by your operating system and we did
     not included it along with the source tarball.  If the function is
     available in Gnulib, it can be fixed immediately.

   • ‘$ make’: _Cannot find the headers (.h files) of installed
     libraries._  Your C preprocessor (CPP) is not looking in the right
     place.  To fix this, configure Gnuastro with an additional
     ‘CPPFLAGS’ like below (assuming the library is installed in
     ‘/usr/local/include’:

          $ ./configure CPPFLAGS="-I/usr/local/include"

     If you want to use the libraries for your other programming
     projects, then export this environment variable in a start-up
     script similar to the case for ‘LD_LIBRARY_PATH’ explained below,
     also see *note Installation directory::.

   • ‘$ make check’: _Only the first couple of tests pass, all the rest
     fail or get skipped._  It is highly likely that when searching for
     shared libraries, your system does not look into the
     ‘/usr/local/lib’ directory (or wherever you installed Gnuastro or
     its dependencies).  To make sure it is added to the list of
     directories, add the following line to your ‘~/.bashrc’ file and
     restart your terminal.  Do Not forget to change ‘/usr/local/lib’ if
     the libraries are installed in other (non-standard) directories.

          export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib"

     You can also add more directories by using a colon ‘‘:’’ to
     separate them.  See *note Installation directory:: and *note
     Linking:: to learn more on the ‘PATH’ variables and dynamic linking
     respectively.

   • ‘$ make check’: _The tests relying on external programs (for
     example, ‘fitstopdf.sh’ fail.)_  This is probably due to the fact
     that the version number of the external programs is too old for the
     tests we have preformed.  Please update the program to a more
     recent version.  For example, to create a PDF image, you will need
     GPL Ghostscript, but older versions do not work, we have
     successfully tested it on version 9.15.  Older versions might cause
     a failure in the test result.

   • ‘$ make pdf’: _The PDF book cannot be made._  To make a PDF book,
     you need to have the GNU Texinfo program (like any program, the
     more recent the better).  A working TeX program is also necessary,
     which you can get from Tex Live(1).

   • After ‘make check’: do not copy the programs’ executables to
     another (for example, the installation) directory manually (using
     ‘cp’, or ‘mv’ for example).  In the default configuration(2), the
     program binaries need to link with Gnuastro’s shared library which
     is also built and installed with the programs.  Therefore, to run
     successfully before and after installation, linking modifications
     need to be made by GNU Libtool at installation time.  ‘make
     install’ does this internally, but a simple copy might give linking
     errors when you run it.  If you need to copy the executables, you
     can do so after installation.

   • ‘$ make’ (when bootstrapping): After you have bootstrapped Gnuastro
     from the version-controlled source, you may confront the following
     (or a similar) error when converting images (for more on
     bootstrapping, see *note Bootstrapping::):

          convert: attempt to perform an operation not allowed by the
          security policy `gs'  error/delegate.c/ExternalDelegateCommand/378.

     This error is a known issue(3) with ‘ImageMagick’ security policies
     in some operating systems.  In short, ‘imagemagick’ uses
     Ghostscript for PDF, EPS, PS and XPS parsing.  However, because
     some security vulnerabilities have been found in Ghostscript(4), by
     default, ImageMagick may be compiled without Ghostscript library.
     In such cases, if allowed, ImageMagick will fall back to the
     external ‘gs’ command instead of the library.  But this may be
     disabled with the following (or a similar) lines in
     ‘/etc/ImageMagick-7/policy.xml’ (anything related to PDF, PS, or
     Ghostscript).

          <policy domain="delegate" rights="none" pattern="gs" />
          <policy domain="module" rights="none" pattern="{PS,PDF,XPS}" />

     To fix this problem, simply comment such lines (by placing a ‘<!--’
     before each statement/line and ‘-->’ at the end of that
     statement/line).

If your problem was not listed above, please file a bug report (*note
Report a bug::).

   ---------- Footnotes ----------

   (1) <https://www.tug.org/texlive/>

   (2) If you configure Gnuastro with the ‘--disable-shared’ option,
then the libraries will be statically linked to the programs and this
problem will not exist, see *note Linking::.

   (3) <https://wiki.archlinux.org/title/ImageMagick>

   (4) <https://security.archlinux.org/package/ghostscript>


File: gnuastro.info,  Node: Common program behavior,  Next: Data containers,  Prev: Installation,  Up: Top

4 Common program behavior
*************************

All the programs in Gnuastro share a set of common behavior mainly to do
with user interaction to facilitate their usage and development.  This
includes how to feed input datasets into the programs, how to configure
them, specifying the outputs, numerical data types, treating columns of
information in tables, etc.  This chapter is devoted to describing this
common behavior in all programs.  Because the behaviors discussed here
are common to several programs, they are not repeated in each program’s
description.

   In *note Command-line::, a very general description of running the
programs on the command-line is discussed, like difference between
arguments and options, as well as options that are common/shared between
all programs.  None of Gnuastro’s programs keep any internal
configuration value (values for their different operational steps), they
read their configuration primarily from the command-line, then from
specific files in directory, user, or system-wide settings.  Using these
configuration files can greatly help reproducible and robust usage of
Gnuastro, see *note Configuration files:: for more.

   It is not possible to always have the different options and
configurations of each program on the top of your head.  It is very
natural to forget the options of a program, their current default
values, or how it should be run and what it did.  Gnuastro’s programs
have multiple ways to help you refresh your memory in multiple levels
(just an option name, a short description, or fast access to the
relevant section of the manual.  See *note Getting help:: for more for
more on benefiting from this very convenient feature.

   Many of the programs use the multi-threaded character of modern CPUs,
in *note Multi-threaded operations:: we will discuss how you can
configure this behavior, along with some tips on making best use of
them.  In *note Numeric data types::, we will review the various types
to store numbers in your datasets: setting the proper type for the usage
context(1) can greatly improve the file size and also speed of reading,
writing or processing them.

   We will then look into the recognized table formats in *note Tables::
and how large datasets are broken into tiles, or mesh grid in *note
Tessellation::.  Finally, we will take a look at the behavior regarding
output files: *note Automatic output:: describes how the programs set a
default name for their output when you do not give one explicitly (using
‘--output’).  When the output is a FITS file, all the programs also
store some very useful information in the header that is discussed in
*note Output FITS files::.

* Menu:

* Command-line::                How to use the command-line.
* Configuration files::         Values for unspecified variables.
* Getting help::                Getting more information on the go.
* Multi-threaded operations::   How threads are managed in Gnuastro.
* Numeric data types::          Different types and how to specify them.
* Memory management::           How memory is allocated (in RAM or HDD/SSD).
* Tables::                      Recognized table formats.
* Tessellation::                Tile the dataset into non-overlapping bins.
* Automatic output::            About automatic output names.
* Output FITS files::           Common properties when outputs are FITS.
* Numeric locale::              Decimal point printed like 0.5 instead of 0,5.

   ---------- Footnotes ----------

   (1) For example, if the values in your dataset can only be integers
between 0 or 65000, store them in a unsigned 16-bit type, not 64-bit
floating point type (which is the default in most systems).  It takes
four times less space and is much faster to process.


File: gnuastro.info,  Node: Command-line,  Next: Configuration files,  Prev: Common program behavior,  Up: Common program behavior

4.1 Command-line
================

Gnuastro’s programs are customized through the standard Unix-like
command-line environment and GNU style command-line options.  Both are
very common in many Unix-like operating system programs.  In *note
Arguments and options:: we will start with the difference between
arguments and options and elaborate on the GNU style of options.
Afterwards, in *note Common options::, we will go into the detailed list
of all the options that are common to all the programs in Gnuastro.

* Menu:

* Arguments and options::       Different ways to specify inputs and configuration.
* Common options::              Options that are shared between all programs.
* Shell TAB completion::        Customized TAB completion in Gnuastro.
* Standard input::              Using output of another program as input.
* Shell tips::                  Useful tips and tricks for program usage.


File: gnuastro.info,  Node: Arguments and options,  Next: Common options,  Prev: Command-line,  Up: Command-line

4.1.1 Arguments and options
---------------------------

When you type a command on the command-line, it is passed onto the shell
(a generic name for the program that manages the command-line) as a
string of characters.  As an example, see the “Invoking ProgramName”
sections in this manual for some examples of commands with each program,
like *note Invoking asttable::, *note Invoking astfits::, or *note
Invoking aststatistics::.

   The shell then brakes up your string into separate _tokens_ or
_words_ using any _metacharacters_ (like white-space, tab, ‘|’, ‘>’ or
‘;’) that are in the string.  On the command-line, the first thing you
usually enter is the name of the program you want to run.  After that,
you can specify two types of tokens: _arguments_ and _options_.  In the
GNU-style, arguments are those tokens that are not preceded by any
hyphens (‘-’, see *note Arguments::).  Here is one example:

     $ astcrop --center=53.162551,-27.789676 -w10/3600 --mode=wcs udf.fits

   In the example above, we are running *note Crop:: to crop a region of
width 10 arc-seconds centered at the given RA and Dec from the input
Hubble Ultra-Deep Field (UDF) FITS image.  Here, the argument is
‘udf.fits’.  Arguments are most commonly the input file names containing
your data.  Options start with one or two hyphens, followed by an
identifier for the option (the option’s name, for example, ‘--center’,
‘-w’, ‘--mode’ in the example above) and its value (anything after the
option name, or the optional <=> character).  Through options you can
configure how the program runs (interprets the data you provided).

   Arguments can be mandatory and optional and unlike options, they do
not have any identifiers.  Hence, when there multiple arguments, their
order might also matter (for example, in ‘cp’ which is used for copying
one file to another location).  The outputs of ‘--usage’ and ‘--help’
shows which arguments are optional and which are mandatory, see *note
--usage::.

   As their name suggests, _options_ can be considered to be optional
and most of the time, you do not have to worry about what order you
specify them in.  When the order does matter, or the option can be
invoked multiple times, it is explicitly mentioned in the “Invoking
ProgramName” section of each program (this is a very important aspect of
an option).

   If there is only one such character, you can use a backslash (‘\’)
before it.  If there are multiple, it might be easier to simply put your
whole argument or option value inside of double quotes (‘"’).  In such
cases, everything inside the double quotes will be seen as one token or
word.

   For example, let’s say you want to specify the header data unit (HDU)
of your FITS file using a complex expression like ‘‘3; images(exposure >
100)’’.  If you simply add these after the ‘--hdu’ (‘-h’) option, the
programs in Gnuastro will read the value to the HDU option as ‘‘3’’ and
run.  Then, the shell will attempt to run a separate command
‘‘images(exposure > 100)’’ and complain about a syntax error.  This is
because the semicolon (‘;’) is an ‘end of command’ character in the
shell.  To solve this problem you can simply put double quotes around
the whole string you want to pass to ‘--hdu’ as seen below:
     $ astcrop --hdu="3; images(exposure > 100)" image.fits

* Menu:

* Arguments::                   For specifying the main input files/operations.
* Options::                     For configuring the behavior of the program.


File: gnuastro.info,  Node: Arguments,  Next: Options,  Prev: Arguments and options,  Up: Arguments and options

4.1.1.1 Arguments
.................

In Gnuastro, arguments are almost exclusively used as the input data
file names.  Please consult the first few paragraph of the “Invoking
ProgramName” section for each program for a description of what it
expects as input, how many arguments, or input data, it accepts, or in
what order.  Everything particular about how a program treats arguments,
is explained under the “Invoking ProgramName” section for that program.

   Generally, if there is a standard file name suffix for a particular
format, that filename extension is checked to identify their format.  In
astronomy (and thus Gnuastro), FITS is the preferred format for inputs
and outputs, so the focus here and throughout this book is on FITS.
However, other formats are also accepted in special cases, for example,
*note ConvertType:: also accepts JPEG or TIFF inputs, and writes JPEG,
EPS or PDF files.  The recognized suffixes for these formats are listed
there.

   The list below shows the recognized suffixes for FITS data files in
Gnuastro’s programs.  However, in some scenarios FITS writers may not
append a suffix to the file, or use a non-recognized suffix (not in the
list below).  Therefore if a FITS file is expected, but it does not have
any of these suffixes, Gnuastro programs will look into the contents of
the file and if it does conform with the FITS standard, the file will be
used.  Just note that checking about 5 characters at the end of a name
string is much more efficient than opening and checking the contents of
a file, so it is generally recommended to have a recognized FITS suffix.

   • ‘.fits’: The standard file name ending of a FITS image.

   • ‘.fit’: Alternative (3 character) FITS suffix.

   • ‘.fits.Z’: A FITS image compressed with ‘compress’.

   • ‘.fits.gz’: A FITS image compressed with GNU zip (gzip).

   • ‘.fits.fz’: A FITS image compressed with ‘fpack’.

   • ‘.imh’: IRAF format image file.

   Through out this book and in the command-line outputs, whenever we
want to generalize all such astronomical data formats in a text
place-holder, we will use ‘ASTRdata’, we will assume that the extension
is also part of this name.  Any file ending with these names is directly
passed on to CFITSIO to read.  Therefore you do not necessarily have to
have these files on your computer, they can also be located on an FTP or
HTTP server too, see the CFITSIO manual for more information.

   CFITSIO has its own error reporting techniques, if your input file(s)
cannot be opened, or read, those errors will be printed prior to the
final error by Gnuastro.


File: gnuastro.info,  Node: Options,  Prev: Arguments,  Up: Arguments and options

4.1.1.2 Options
...............

Command-line options allow configuring the behavior of a program in all
GNU/Linux applications for each particular execution on a particular
input data.  A single option can be called in two ways: _long_ or
_short_.  All options in Gnuastro accept the long format which has two
hyphens an can have many characters (for example, ‘--hdu’).  Short
options only have one hyphen (<->) followed by one character (for
example, ‘-h’).  You can see some examples in the list of options in
*note Common options:: or those for each program’s “Invoking
ProgramName” section.  Both formats are shown for those which support
both.  First the short is shown then the long.

   Usually, the short options are for when you are writing on the
command-line and want to save keystrokes and time.  The long options are
good for shell scripts, where you are not usually rushing.  Long options
provide a level of documentation, since they are more descriptive and
less cryptic.  Usually after a few months of not running a program, the
short options will be forgotten and reading your previously written
script will not be easy.

   Some options need to be given a value if they are called and some do
not.  You can think of the latter type of options as on/off options.
These two types of options can be distinguished using the output of the
‘--help’ and ‘--usage’ options, which are common to all GNU software,
see *note Getting help::.  In Gnuastro we use the following strings to
specify when the option needs a value and what format that value should
be in.  More specific tests will be done in the program and if the
values are out of range (for example, negative when the program only
wants a positive value), an error will be reported.

‘INT’
     The value is read as an integer.

‘FLT’
     The value is read as a float.  There are generally two types,
     depending on the context.  If they are for fractions, they will
     have to be less than or equal to unity.

‘STR’
     The value is read as a string of characters.  For example, column
     names in a table, or HDU names in a multi-extension FITS file.
     Other examples include human-readable settings by some programs
     like the ‘--domain’ option of the Convolve program that can be
     either ‘spatial’ or ‘frequency’ (to specify the type of
     convolution, see *note Convolve::).

‘FITS or FITS/TXT’
     The value should be a file (most commonly FITS). In many cases,
     other formats may also be accepted (for example, input tables can
     be FITS or plain-text, see *note Recognized table formats::).

To specify a value in the short format, simply put the value after the
option.  Note that since the short options are only one character long,
you do not have to type anything between the option and its value.  For
the long option you either need white space or an ‘=’ sign, for example,
‘-h2’, ‘-h 2’, ‘--hdu 2’ or ‘--hdu=2’ are all equivalent.

   The short format of on/off options (those that do not need values)
can be concatenated for example, these two hypothetical sequences of
options are equivalent: ‘-a -b -c4’ and ‘-abc4’.  As an example,
consider the following command to run Crop:
     $ astcrop -Dr3 --wwidth 3 catalog.txt --deccol=4 ASTRdata
The ‘$’ is the shell prompt, ‘astcrop’ is the program name.  There are
two arguments (‘catalog.txt’ and ‘ASTRdata’) and four options, two of
them given in short format (‘-D’, ‘-r’) and two in long format
(‘--width’ and ‘--deccol’).  Three of them require a value and one
(‘-D’) is an on/off option.

   If an abbreviation is unique between all the options of a program,
the long option names can be abbreviated.  For example, instead of
typing ‘--printparams’, typing ‘--print’ or maybe even ‘--pri’ will be
enough, if there are conflicts, the program will warn you and show you
the alternatives.  Finally, if you want the argument parser to stop
parsing arguments beyond a certain point, you can use two dashes: ‘--’.
No text on the command-line beyond these two dashes will be parsed.

   Gnuastro has two types of options with values, those that only take a
single value are the most common type.  If these options are repeated or
called more than once on the command-line, the value of the last time it
was called will be assigned to it.  This is very useful when you are
testing/experimenting.  Let’s say you want to make a small modification
to one option value.  You can simply type the option with a new value in
the end of the command and see how the script works.  If you are
satisfied with the change, you can remove the original option for human
readability.  If the change was not satisfactory, you can remove the one
you just added and not worry about forgetting the original value.
Without this capability, you would have to memorize or save the original
value somewhere else, run the command and then change the value again
which is not at all convenient and is potentially cause lots of bugs.

   On the other hand, some options can be called multiple times in one
run of a program and can thus take multiple values (for example, see the
‘--column’ option in *note Invoking asttable::.  In these cases, the
order of stored values is the same order that you specified on the
command-line.

   Gnuastro’s programs do not keep any internal default values, so some
options are mandatory and if they do not have a value, the program will
complain and abort.  Most programs have many such options and typing
them by hand on every call is impractical.  To facilitate the user
experience, after parsing the command-line, Gnuastro’s programs read
special configuration files to get the necessary values for the options
you have not identified on the command-line.  These configuration files
are fully described in *note Configuration files::.

*CAUTION:* In specifying a file address, if you want to use the shell’s
tilde expansion (‘~’) to specify your home directory, leave at least one
space between the option name and your value.  For example, use ‘-o
~/test’, ‘--output ~/test’ or ‘--output= ~/test’.  Calling them with
‘-o~/test’ or ‘--output=~/test’ will disable shell expansion.
*CAUTION:* If you forget to specify a value for an option which requires
one, and that option is the last one, Gnuastro will warn you.  But if it
is in the middle of the command, it will take the text of the next
option or argument as the value which can cause undefined behavior.
*NOTE:* In some contexts Gnuastro’s counting starts from 0 and in others
1.  You can assume by default that counting starts from 1, if it starts
from 0 for a special option, it will be explicitly mentioned.


File: gnuastro.info,  Node: Common options,  Next: Shell TAB completion,  Prev: Arguments and options,  Up: Command-line

4.1.2 Common options
--------------------

To facilitate the job of the users and developers, all the programs in
Gnuastro share some basic command-line options for the options that are
common to many of the programs.  The full list is classified as *note
Input output options::, *note Processing options::, and *note Operating
mode options::.  In some programs, some of the options are irrelevant,
but still recognized (you will not get an unrecognized option error, but
the value is not used).  Unless otherwise mentioned, these options are
identical between all programs.

* Menu:

* Input output options::        Common input/output options.
* Processing options::          Options for common processing steps.
* Operating mode options::      Common operating mode options.


File: gnuastro.info,  Node: Input output options,  Next: Processing options,  Prev: Common options,  Up: Common options

4.1.2.1 Input/Output options
............................

These options are to do with the input and outputs of the various
programs.

‘--stdintimeout’
     Number of micro-seconds to wait for writing/typing in the _first
     line_ of standard input from the command-line (see *note Standard
     input::).  This is only relevant for programs that also accept
     input from the standard input, _and_ you want to manually
     write/type the contents on the terminal.  When the standard input
     is already connected to a pipe (output of another program), there
     will not be any waiting (hence no timeout, thus making this option
     redundant).

     If the first line-break (for example, with the <ENTER> key) is not
     provided before the timeout, the program will abort with an error
     that no input was given.  Note that this time interval is _only_
     for the first line that you type.  Once the first line is given,
     the program will assume that more data will come and accept rest of
     your inputs without any time limit.  You need to specify the ending
     of the standard input, for example, by pressing <CTRL-D> after a
     new line.

     Note that any input you write/type into a program on the
     command-line with Standard input will be discarded (lost) once the
     program is finished.  It is only recoverable manually from your
     command-line (where you actually typed) as long as the terminal is
     open.  So only use this feature when you are sure that you do not
     need the dataset (or have a copy of it somewhere else).

‘-h STR/INT’
‘--hdu=STR/INT’
     The name or number of the desired Header Data Unit, or HDU, in the
     FITS image.  A FITS file can store multiple HDUs or extensions,
     each with either an image or a table or nothing at all (only a
     header).  Note that counting of the extensions starts from 0(zero),
     not 1(one).  Counting from 0 is forced on us by CFITSIO which
     directly reads the value you give with this option (see *note
     CFITSIO::).  When specifying the name, case is not important so
     ‘IMAGE’, ‘image’ or ‘ImAgE’ are equivalent.

     CFITSIO has many capabilities to help you find the extension you
     want, far beyond the simple extension number and name.  See CFITSIO
     manual’s “HDU Location Specification” section for a very complete
     explanation with several examples.  A ‘#’ is appended to the string
     you specify for the HDU(1) and the result is put in square brackets
     and appended to the FITS file name before calling CFITSIO to read
     the contents of the HDU for all the programs in Gnuastro.

     *Default HDU is HDU number 1 (counting from 0):* by default,
     Gnuastro’s programs assume that their (main/first) input is in HDU
     number 1 (counting from zero).  So if you don’t specify the HDU
     number, the program will read the input from this HDU. For programs
     that can take multiple FITS datasets as input (like *note
     Arithmetic::) this default HDU applies to the first input, you
     still need to call ‘--hdu’ for the other inputs.  Generally, all
     Gnuastro’s programs write their outputs in HDU number 1 (HDU 0 is
     reserved for metadata like the configuration parameters that the
     program was run with).  For more on this, see *note Fits::.

‘-s STR’
‘--searchin=STR’
     Where to match/search for columns when the column identifier was
     not a number, see *note Selecting table columns::.  The acceptable
     values are ‘name’, ‘unit’, or ‘comment’.  This option is only
     relevant for programs that take table columns as input.

‘-I’
‘--ignorecase’
     Ignore case while matching/searching column meta-data (in the field
     specified by the ‘--searchin’).  The FITS standard suggests to
     treat the column names as case insensitive, which is strongly
     recommended here also but is not enforced.  This option is only
     relevant for programs that take table columns as input.

     This option is not relevant to *note BuildProgram::, hence in that
     program the short option ‘-I’ is used for include directories, not
     to ignore case.

‘-o STR’
‘--output=STR’
     The name of the output file or directory.  With this option the
     automatic output names explained in *note Automatic output:: are
     ignored.

‘-T STR’
‘--type=STR’
     The data type of the output depending on the program context.  This
     option is not applicable to some programs like *note Fits:: and
     will be ignored by them.  The different acceptable values to this
     option are fully described in *note Numeric data types::.

‘-D’
‘--dontdelete’
     By default, if the output file already exists, Gnuastro’s programs
     will silently delete it and put their own outputs in its place.
     When this option is activated, if the output file already exists,
     the programs will not delete it, will warn you, and will abort.

‘-K’
‘--keepinputdir’
     In automatic output names, do not remove the directory information
     of the input file names.  As explained in *note Automatic output::,
     if no output name is specified (with ‘--output’), then the output
     name will be made in the existing directory based on your input’s
     file name (ignoring the directory of the input).  If you call this
     option, the directory information of the input will be kept and the
     automatically generated output name will be in the same directory
     as the input (usually with a suffix added).  Note that his is only
     relevant if you are running the program in a different directory
     than the input data.

‘-t STR’
‘--tableformat=STR’
     The output table’s type.  This option is only relevant when the
     output is a table and its format cannot be deduced from its
     filename.  For example, if a name ending in ‘.fits’ was given to
     ‘--output’, then the program knows you want a FITS table.  But
     there are two types of FITS tables: FITS ASCII, and FITS binary.
     Thus, with this option, the program is able to identify which type
     you want.  The currently recognized values to this option are:

‘--wcslinearmatrix=STR’
     Select the linear transformation matrix of the output’s WCS. This
     option only takes two values: ‘pc’ (for the ‘PCi_j’ formalism) and
     ‘cd’ (for ‘CDi_j’).  For more on the different formalisms, please
     see Section 8.1 of the FITS standard(2), version 4.0.

     In short, in the ‘PCi_j’ formalism, we only keep the linear
     rotation matrix in these keywords and put the scaling factor (or
     the pixel scale in astronomical imaging) in the ‘CDELTi’ keywords.
     In the ‘CDi_j’ formalism, we blend the scaling into the rotation
     into a single matrix and keep that matrix in these FITS keywords.
     By default, Gnuastro uses the ‘PCi_j’ formalism, because it greatly
     helps in human readability of the raw keywords and is also the
     default mode of WCSLIB. However, in some circumstances it may be
     necessary to have the keywords in the CD format; for example, when
     you need to feed the outputs into other software that do not follow
     the full FITS standard and only recognize the ‘CDi_j’ formalism.

     ‘txt’
          A plain text table with white-space characters between the
          columns (see *note Gnuastro text table format::).
     ‘fits-ascii’
          A FITS ASCII table (see *note Recognized table formats::).
     ‘fits-binary’
          A FITS binary table (see *note Recognized table formats::).

   ---------- Footnotes ----------

   (1) With the ‘#’ character, CFITSIO will only read the desired HDU
into your memory, not all the existing HDUs in the fits file.

   (2) <https://fits.gsfc.nasa.gov/standard40/fits_standard40aa-le.pdf>


File: gnuastro.info,  Node: Processing options,  Next: Operating mode options,  Prev: Input output options,  Up: Common options

4.1.2.2 Processing options
..........................

Some processing steps are common to several programs, so they are
defined as common options to all programs.  Note that this class of
common options is thus necessarily less common between all the programs
than those described in *note Input output options::, or *note Operating
mode options:: options.  Also, if they are irrelevant for a program,
these options will not display in the ‘--help’ output of the program.

‘--minmapsize=INT’
     The minimum size (in bytes) to memory-map a processing/internal
     array as a file (on the non-volatile HDD/SSD), and not use the
     system’s RAM. Before using this option, please read *note Memory
     management::.  By default processing arrays will only be
     memory-mapped to a file when the RAM is full.  With this option,
     you can force the memory-mapping, even when there is enough RAM. To
     ensure this default behavior, the pre-defined value to this option
     is an extremely large value (larger than any existing RAM).

     Please note that using a non-volatile file (in the HDD/SDD) instead
     of RAM can significantly increase the program’s running time,
     especially on HDDs (where read/write is slower).  Also, note that
     the number of memory-mapped files that your kernel can support is
     limited.  So when this option is necessary, it is best to give it
     values larger than 1 megabyte (‘--minmapsize=1000000’).  You can
     then decrease it for a specific program’s invocation on a large
     input after you see memory issues arise (for example, an error, or
     the program not aborting and fully consuming your memory).  If you
     see randomly named files remaining in this directory when the
     program finishes normally, please send us a bug report so we
     address the problem, see *note Report a bug::.

     *Limited number of memory-mapped files:* The operating system
     kernels usually support a limited number of memory-mapped files.
     Therefore never set ‘--minmapsize’ to zero or a small number of
     bytes (so too many files are created).  If the kernel capacity is
     exceeded, the program will crash.

‘--quietmmap’
     Do Not print any message when an array is stored in non-volatile
     memory (HDD/SSD) and not RAM, see the description of ‘--minmapsize’
     (above) for more.

‘-Z INT[,INT[,...]]’
‘--tilesize=[,INT[,...]]’
     The size of regular tiles for tessellation, see *note
     Tessellation::.  For each dimension an integer length (in units of
     data-elements or pixels) is necessary.  If the number of input
     dimensions is different from the number of values given to this
     option, the program will stop with an error.  Values must be
     separated by commas (<,>) and can also be fractions (for example,
     ‘4/2’).  If they are fractions, the result must be an integer,
     otherwise an error will be printed.

‘-M INT[,INT[,...]]’
‘--numchannels=INT[,INT[,...]]’
     The number of channels for larger input tessellation, see *note
     Tessellation::.  The number and types of acceptable values are
     similar to ‘--tilesize’.  The only difference is that instead of
     length, the integers values given to this option represent the
     _number_ of channels, not their size.

‘-F FLT’
‘--remainderfrac=FLT’
     The fraction of remainder size along all dimensions to add to the
     first tile.  See *note Tessellation:: for a complete description.
     This option is only relevant if ‘--tilesize’ is not exactly
     divisible by the input dataset’s size in a dimension.  If the
     remainder size is larger than this fraction (compared to
     ‘--tilesize’), then the remainder size will be added with one
     regular tile size and divided between two tiles at the start and
     end of the given dimension.

‘--workoverch’
     Ignore the channel borders for the high-level job of the given
     application.  As a result, while the channel borders are respected
     in defining the small tiles (such that no tile will cross a channel
     border), the higher-level program operation will ignore them, see
     *note Tessellation::.

‘--checktiles’
     Make a FITS file with the same dimensions as the input but each
     pixel is replaced with the ID of the tile that it is associated
     with.  Note that the tile IDs start from 0.  See *note
     Tessellation:: for more on Tiling an image in Gnuastro.

‘--oneelempertile’
     When showing the tile values (for example, with ‘--checktiles’, or
     when the program’s output is tessellated) only use one element for
     each tile.  This can be useful when only the relative values given
     to each tile compared to the rest are important or need to be
     checked.  Since the tiles usually have a large number of pixels
     within them the output will be much smaller, and so easier to read,
     write, store, or send.

     Note that when the full input size in any dimension is not exactly
     divisible by the given ‘--tilesize’ in that dimension, the edge
     tile(s) will have different sizes (in units of the input’s size),
     see ‘--remainderfrac’.  But with this option, all displayed values
     are going to have the (same) size of one data-element.  Hence, in
     such cases, the image proportions are going to be slightly
     different with this option.

     If your input image is not exactly divisible by the tile size and
     you want one value per tile for some higher-level processing, all
     is not lost though.  You can see how many pixels were within each
     tile (for example, to weight the values or discard some for later
     processing) with Gnuastro’s Statistics (see *note Statistics::) as
     shown below.  The output FITS file is going to have two extensions,
     one with the median calculated on each tile and one with the number
     of elements that each tile covers.  You can then use the ‘where’
     operator in *note Arithmetic:: to set the values of all tiles that
     do not have the regular area to a blank value.

          $ aststatistics --median --number --ontile input.fits    \
                          --oneelempertile --output=o.fits
          $ REGULAR_AREA=1600    # Check second extension of `o.fits'.
          $ astarithmetic o.fits o.fits $REGULAR_AREA ne nan where \
                          -h1 -h2

     Note that if ‘input.fits’ also has blank values, then the median on
     tiles with blank values will also be ignored with the command above
     (which is desirable).

‘--inteponlyblank’
     When values are to be interpolated, only change the values of the
     blank elements, keep the non-blank elements untouched.

‘--interpmetric=STR’
     The metric to use for finding nearest neighbors.  Currently it only
     accepts the Manhattan (or taxicab) metric with ‘manhattan’, or the
     radial metric with ‘radial’.

     The Manhattan distance between two points is defined with
     $|\Delta{x}|+|\Delta{y}|$.  Thus the Manhattan metric has the
     advantage of being fast, but at the expense of being less accurate.
     The radial distance is the standard definition of distance in a
     Euclidean space: $\sqrt{\Delta{x}^2+\Delta{y}^2}$.  It is accurate,
     but the multiplication and square root can slow down the
     processing.

‘--interpnumngb=INT’
     The number of nearby non-blank neighbors to use for interpolation.


File: gnuastro.info,  Node: Operating mode options,  Prev: Processing options,  Up: Common options

4.1.2.3 Operating mode options
..............................

Another group of options that are common to all the programs in Gnuastro
are those to do with the general operation of the programs.  The
explanation for those that are not only limited to Gnuastro but are
common to all GNU programs start with (GNU option).

‘--’
     (GNU option) Stop parsing the command-line.  This option can be
     useful in scripts or when using the shell history.  Suppose you
     have a long list of options, and want to see if removing some of
     them (to read from configuration files, see *note Configuration
     files::) can give a better result.  If the ones you want to remove
     are the last ones on the command-line, you do not have to delete
     them, you can just add ‘--’ before them and if you do not get what
     you want, you can remove the ‘--’ and get the same initial result.

‘--usage’
     (GNU option) Only print the options and arguments and abort.  This
     is very useful for when you know the what the options do, and have
     just forgot their long/short identifiers, see *note --usage::.

‘-?’
‘--help’
     (GNU option) Print all options with an explanation and abort.
     Adding this option will print all the options in their short and
     long formats, also displaying which ones need a value if they are
     called (with an ‘=’ after the long format followed by a string
     specifying the format, see *note Options::).  A short explanation
     is also given for what the option is for.  The program will quit
     immediately after the message is printed and will not do any form
     of processing, see *note --help::.

‘-V’
‘--version’
     (GNU option) Print a short message, showing the full name, version,
     copyright information and program authors and abort.  On the first
     line, it will print the official name (not executable name) and
     version number of the program.  Following this is a blank line and
     a copyright information.  The program will not run.

‘-q’
‘--quiet’
     Do Not report steps.  All the programs in Gnuastro that have
     multiple major steps will report their steps for you to follow
     while they are operating.  If you do not want to see these reports,
     you can call this option and only error/warning messages will be
     printed.  If the steps are done very fast (depending on the
     properties of your input) disabling these reports will also
     decrease running time.

‘--cite’
     Print all necessary information to cite and acknowledge Gnuastro in
     your published papers.  With this option, the programs will print
     the BibTeX entry to include in your paper for Gnuastro in general,
     and the particular program’s paper (if that program comes with a
     separate paper).  It will also print the necessary acknowledgment
     statement to add in the respective section of your paper and it
     will abort.  For a more complete explanation, please see *note
     Acknowledgments::.

     Citations and acknowledgments are vital for the continued work on
     Gnuastro.  Gnuastro started, and is continued, based on separate
     research projects.  So if you find any of the tools offered in
     Gnuastro to be useful in your research, please use the output of
     this command to cite and acknowledge the program (and Gnuastro) in
     your research paper.  Thank you.

     Gnuastro is still new, there is no separate paper only devoted to
     Gnuastro yet.  Therefore currently the paper to cite for Gnuastro
     is the paper for NoiseChisel which is the first published paper
     introducing Gnuastro to the astronomical community.  Upon reaching
     a certain point, a paper completely devoted to describing
     Gnuastro’s many functionalities will be published, see *note GNU
     Astronomy Utilities 1.0::.

‘-P’
‘--printparams’
     With this option, Gnuastro’s programs will read your command-line
     options and all the configuration files.  If there is no problem
     (like a missing parameter or a value in the wrong format or range)
     and immediately before actually running, the programs will print
     the full list of option names, values and descriptions, sorted and
     grouped by context and abort.  They will also report the version
     number, the date they were configured on your system and the time
     they were reported.

     As an example, you can give your full command-line options and even
     the input and output file names and finally just add ‘-P’ to check
     if all the parameters are finely set.  If everything is OK, you can
     just run the same command (easily retrieved from the shell history,
     with the top arrow key) and simply remove the last two characters
     that showed this option.

     No program will actually start its processing when this option is
     called.  The otherwise mandatory arguments for each program (for
     example, input image or catalog files) are no longer required when
     you call this option.

‘--config=STR’
     Parse ‘STR’ as a configuration file name, immediately when this
     option is confronted (see *note Configuration files::).  The
     ‘--config’ option can be called multiple times in one run of any
     Gnuastro program on the command-line or in the configuration files.
     In any case, it will be immediately read (before parsing the rest
     of the options on the command-line, or lines in a configuration
     file).  If the given file does not exist or cannot be read for any
     reason, the program will print a warning and continue its
     processing.  The warning can be suppressed with ‘--quiet’.

     Note that by definition, options on the command-line still take
     precedence over those in any configuration file, including the
     file(s) given to this option if they are called before it.  Also
     see ‘--lastconfig’ and ‘--onlyversion’ on how this option can be
     used for reproducible results.  You can use ‘--checkconfig’ (below)
     to check/confirm the parsing of configuration files.

‘--checkconfig’
     Print options and their values, within the command-line or
     configuration files, as they are parsed (see *note Configuration
     file precedence::).  If an option has already been set, or is
     ignored by the program, this option will also inform you with
     special values like ‘--ALREADY-SET--’.  Only options that are
     parsed after this option are printed, so to see the parsing of all
     input options, it is recommended to put this option immediately
     after the program name before any other options.

     This is a very good option to confirm where the value of each
     option is has been defined in scenarios where there are multiple
     configuration files (for debugging).

‘--config-prefix=STR’
     Accept option names in configuration files that start with the
     given prefix.  Since order matters when reading custom
     configuration files, this option should be called *before* the
     ‘--config’ option(s) that contain options with the given prefix.
     This option does not affect the options within configuration files
     that have the standard name (without a prefix).

     This gives unique features to Gnuastro’s configuration files,
     especially in large pipelines.  Let’s demonstrate this with the
     simple scenario below.  You have multiple configuration files for
     different instances of one program (let’s assume ‘nc-a.conf’ and
     ‘nc-b.conf’).  At the same time, want to load all the option
     names/values into your shell as environment variables (for example
     with ‘source’).  This happens when you want to use the options as
     shell variables in other parts of the your pipeline.

     If the two configuration files have different values for the same
     option (as shown below), and you don’t use ‘--config-prefix’, the
     shell will over-write the common option values between the
     configuration files.  But thanks to ‘--config-prefix’, you can give
     a different prefix for the different instances of the same option
     in different configuration files.

          $ cat nc-a.conf
          a_tilesize=20,20

          $ cat nc-b.conf
          b_tilesize=40,40

          ## Load configuration files as shell scripts (to define the
          ## option name and values as shell variables with values).
          ## Just note that 'source' only takes one file at a time.
          $ for c in nc-*.conf; do source $c; done

          $ astnoisechisel img.fits \
                           --config=nc-a.conf --config-prefix=a_
          $ echo "NoiseChisel run with --tilesize=$a_tilesize"

          $ astnoisechisel img.fits \
                           --config=nc-b.conf --config-prefix=b_
          $ echo "NoiseChisel run with --tilesize=$b_tilesize"

‘-S’
‘--setdirconf’
     Update the current directory configuration file for the Gnuastro
     program and quit.  The full set of command-line and configuration
     file options will be parsed and options with a value will be
     written in the current directory configuration file for this
     program (see *note Configuration files::).  If the configuration
     file or its directory does not exist, it will be created.  If a
     configuration file exists it will be replaced (after it, and all
     other configuration files have been read).  In any case, the
     program will not run.

     This is the recommended method(1) to edit/set the configuration
     file for all future calls to Gnuastro’s programs.  It will
     internally check if your values are in the correct range and type
     and save them according to the configuration file format, see *note
     Configuration file format::.  So if there are unreasonable values
     to some options, the program will notify you and abort before
     writing the final configuration file.

     When this option is called, the otherwise mandatory arguments, for
     example input image or catalog file(s), are no longer mandatory
     (since the program will not run).

‘-U’
‘--setusrconf’
     Update the user configuration file and quit (see *note
     Configuration files::).  See explanation under ‘--setdirconf’ for
     more details.

‘--lastconfig’
     This is the last configuration file that must be read.  When this
     option is confronted in any stage of reading the options (on the
     command-line or in a configuration file), no other configuration
     file will be parsed, see *note Configuration file precedence:: and
     *note Current directory and User wide::.  Like all on/off options,
     on the command-line, this option does not take any values.  But in
     a configuration file, it takes the values of ‘0’ or ‘1’, see *note
     Configuration file format::.  If it is present in a configuration
     file with a value of ‘0’, then all later occurrences of this option
     will be ignored.

‘--onlyversion=STR’
     Only run the program if Gnuastro’s version is exactly equal to
     ‘STR’ (see *note Version numbering::).  Note that it is not
     compared as a number, but as a string of characters, so ‘0’, or
     ‘0.0’ and ‘0.00’ are different.  If the running Gnuastro version is
     different, then this option will report an error and abort as soon
     as it is confronted on the command-line or in a configuration file.
     If the running Gnuastro version is the same as ‘STR’, then the
     program will run as if this option was not called.

     This is useful if you want your results to be exactly reproducible
     and not mistakenly run with an updated/newer or older version of
     the program.  Besides internal algorithmic/behavior changes in
     programs, the existence of options or their names might change
     between versions (especially in these earlier versions of
     Gnuastro).

     Hence, when using this option (probably in a script or in a
     configuration file), be sure to call it before other options.  The
     benefit is that, when the version differs, the other options will
     not be parsed and you, or your collaborators/users, will not get
     errors saying an option in your configuration does not exist in the
     running version of the program.

     Here is one example of how this option can be used in conjunction
     with the ‘--lastconfig’ option.  Let’s assume that you were
     satisfied with the results of this command: ‘astnoisechisel
     image.fits --snquant=0.95’ (along with various options set in
     various configuration files).  You can save the state of
     NoiseChisel and reproduce that exact result on ‘image.fits’ later
     by following these steps (the extra spaces, and <\>, are only for
     easy readability, if you want to try it out, only one space between
     each token is enough).

          $ echo "onlyversion X.XX"             > reproducible.conf
          $ echo "lastconfig 1"                >> reproducible.conf
          $ astnoisechisel image.fits --snquant=0.95 -P            \
                                               >> reproducible.conf

     ‘--onlyversion’ was available from Gnuastro 0.0, so putting it
     immediately at the start of a configuration file will ensure that
     later, you (or others using different version) will not get a
     non-recognized option error in case an option was added/removed.
     ‘--lastconfig’ will inform the installed NoiseChisel to not parse
     any other configuration files.  This is done because we do not want
     the user’s user-wide or system wide option values affecting our
     results.  Finally, with the third command, which has a ‘-P’ (short
     for ‘--printparams’), NoiseChisel will print all the option values
     visible to it (in all the configuration files) and the shell will
     append them to ‘reproduce.conf’.  Hence, you do not have to worry
     about remembering the (possibly) different options in the different
     configuration files.

     Afterwards, if you run NoiseChisel as shown below (telling it to
     read this configuration file with the ‘--config’ option).  You can
     be sure that there will either be an error (for version mismatch)
     or it will produce exactly the same result that you got before.

          $ astnoisechisel --config=reproducible.conf

‘--log’
     Some programs can generate extra information about their outputs in
     a log file.  When this option is called in those programs, the log
     file will also be printed.  If the program does not generate a log
     file, this option is ignored.

     *‘--log’ is not thread-safe*: The log file usually has a fixed
     name.  Therefore if two simultaneous calls (with ‘--log’) of a
     program are made in the same directory, the program will try to
     write to he same file.  This will cause problems like unreasonable
     log file, undefined behavior, or a crash.

‘-N INT’
‘--numthreads=INT’
     Use ‘INT’ CPU threads when running a Gnuastro program (see *note
     Multi-threaded operations::).  If the value is zero (‘0’), or this
     option is not given on the command-line or any configuration file,
     the value will be determined at run-time: the maximum number of
     threads available to the system when you run a Gnuastro program.

     Note that multi-threaded programming is only relevant to some
     programs.  In others, this option will be ignored.

   ---------- Footnotes ----------

   (1) Alternatively, you can use your favorite text editor.


File: gnuastro.info,  Node: Shell TAB completion,  Next: Standard input,  Prev: Common options,  Up: Command-line

4.1.3 Shell TAB completion (highly customized)
----------------------------------------------

*Under development:* Gnuastro’s TAB completion in Bash already greatly
improves usage of Gnuastro on the command-line, but still under
development and not yet complete.  If you are interested to try it out,
please go ahead and activate it (as described below), we encourage this.
But please have in mind that there are known issues(1) and you may find
new issues.  If you do, please get in touch with us as described in
*note Report a bug::.  TAB completion is currently only implemented in
the following programs: Arithmetic, BuildProgram, ConvertType, Convolve,
CosmicCalculator, Crop, Fits and Table.  For progress on this task,
please see Task 15799(2).

   Bash provides a built-in feature called _programmable completion_(3)
to help increase interactive workflow efficiency and minimize the number
of key-strokes _and_ the need to memorize things.  It is also known as
TAB completion, bash completion, auto-completion, or word completion.
Completion is activated by pressing <[TAB]> while you are typing a
command.  For file arguments this is the default behavior already and
you have probably used it a lot with any command-line program.

   Besides this simple/default mode, Bash also enables a high level of
customization features for its completion.  These features have been
extensively used in Gnuastro to improve your work efficiency(4).  For
example, if you are running ‘asttable’ (which only accepts files
containing a table), and you press <[TAB]>, it will only suggest files
containing tables.  As another example, if an option needs image HDUs
within a FITS file, pressing <[TAB]> will only suggest the image HDUs
(and not other possibly existing HDUs that contain tables, or just
metadata).  Just note that the file name has to be already given on the
command-line before reaching such options (that look into the contents
of a file).

   But TAB completion is not limited to file types or contents.
Arguments/Options that take certain fixed string values will directly
suggest those strings with TAB, and completely ignore the file structure
(for example, spectral line names in *note Invoking astcosmiccal::)!  As
another example, the option ‘--numthreads’ option (to specify the number
of threads to use by the program), will find the number of available
threads on the system, and suggest the possible numbers with a TAB!

   To activate Gnuastro’s custom TAB completion in Bash, you need to put
the following line in one of your Bash startup files (for example,
‘~/.bashrc’).  If you installed Gnuastro using the steps of *note Quick
start::, you should have already done this (the command just after ‘sudo
make install’).  For a list of (and discussion on) Bash startup files
and installation directories see *note Installation directory::.  Of
course, if Gnuastro was installed in a custom location, replace the
‘‘/usr/local’’ part of the line below to the value that was given to
‘--prefix’ during Gnuastro’s configuration(5).

     # Enable Gnuastro's TAB completion
     source /usr/local/share/gnuastro/completion.bash

   After adding the line above in a Bash startup file, TAB completion
will always be activated in any new terminal.  To see if it has been
activated, try it out with ‘asttable [TAB][TAB]’ and ‘astarithmetic
[TAB][TAB]’ in a directory that contains tables and images.  The first
will only suggest the files with a table, and the second, only those
with an image.

*TAB completion only works with long option names:* As described above,
short options are much more complex to generalize, therefore TAB
completion is only available for long options.  But do not worry!  TAB
completion also involves option names, so if you just type
‘--a[TAB][TAB]’, you will get the list of options that start with an
‘--a’.  Therefore as a side-effect of TAB completion, your commands will
be far more human-readable with minimal key strokes.

   ---------- Footnotes ----------

   (1) 
<http://savannah.gnu.org/bugs/index.php?group=gnuastro&category_id=128>

   (2) <https://savannah.gnu.org/task/?15799>

   (3) 
<https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion.html>

   (4) To learn how Gnuastro implements TAB completion in Bash, see
*note Bash programmable completion::.

   (5) In case you do not know the installation directory of Gnuastro on
your system, you can find out with this command: ‘which astfits | sed
-e"s|/bin/astfits||"’


File: gnuastro.info,  Node: Standard input,  Next: Shell tips,  Prev: Shell TAB completion,  Up: Command-line

4.1.4 Standard input
--------------------

The most common way to feed the primary/first input dataset into a
program is to give its filename as an argument (discussed in *note
Arguments::).  When you want to run a series of programs in sequence,
this means that each will have to keep the output of each program in a
separate file and re-type that file’s name in the next command.  This
can be very slow and frustrating (mis-typing a file’s name).

   To solve the problem, the founders of Unix defined pipes to directly
feed the output of one program (its “Standard output” stream) into the
“standard input” of a next program.  This removes the need to make
temporary files between separate processes and became one of the best
demonstrations of the Unix-way, or Unix philosophy.

   Every program has three streams identifying where it reads/writes
non-file inputs/outputs: _Standard input_, _Standard output_, and
_Standard error_.  When a program is called alone, all three are
directed to the terminal that you are using.  If it needs an input, it
will prompt you for one and you can type it in.  Or, it prints its
results in the terminal for you to see.

   For example, say you have a FITS table/catalog containing the B and V
band magnitudes (‘MAG_B’ and ‘MAG_V’ columns) of a selection of galaxies
along with many other columns.  If you want to see only these two
columns in your terminal, can use Gnuastro’s *note Table:: program like
below:

     $ asttable cat.fits -cMAG_B,MAG_V

   Through the Unix pipe mechanism, when the shell confronts the pipe
character (<|>), it connects the standard output of the program before
the pipe, to the standard input of the program after it.  So it is
literally a “pipe”: everything that you would see printed by the first
program on the command (without any pipe), is now passed to the second
program (and not seen by you).

   To continue the previous example, let’s say you want to see the B-V
color.  To do this, you can pipe Table’s output to AWK (a wonderful tool
for processing things like plain text tables):

     $ asttable cat.fits -cMAG_B,MAG_V | awk '{print $1-$2}'

   But understanding the distribution by visually seeing all the numbers
under each other is not too useful!  You can therefore feed this single
column information into *note Statistics:: to give you a general feeling
of the distribution with the same command:

     $ asttable cat.fits -cMAG_B,MAG_V | awk '{print $1-$2}' | aststatistics

   Gnuastro’s programs that accept input from standard input, only look
into the Standard input stream if there is no first argument.  In other
words, arguments take precedence over Standard input.  When no argument
is provided, the programs check if the standard input stream is already
full or not (output from another program is waiting to be used).  If
data is present in the standard input stream, it is used.

   When the standard input is empty, the program will wait
‘--stdintimeout’ micro-seconds for you to manually enter the first line
(ending with a new-line character, or the <ENTER> key, see *note Input
output options::).  If it detects the first line in this time, there is
no more time limit, and you can manually write/type all the lines for as
long as it takes.  To inform the program that Standard input has
finished, press <CTRL-D> after a new line.  If the program does not
catch the first line before the time-out finishes, it will abort with an
error saying that no input was provided.

*Manual input in Standard input is discarded:* Be careful that when you
manually fill the Standard input, the data will be discarded once the
program finishes and reproducing the result will be impossible.
Therefore this form of providing input is only good for temporary tests.

*Standard input currently only for plain text:* Currently Standard input
only works for plain text inputs like the example above.  We will later
allow FITS files into the programs through standard input also.


File: gnuastro.info,  Node: Shell tips,  Prev: Standard input,  Up: Command-line

4.1.5 Shell tips
----------------

Gnuastro’s programs are primarily meant to be run on the command-line
shell environment.  In this section, we will review some useful tips and
tricks that can be helpful in the pipelines that you run.

* Menu:

* Separate shell variables for multiple outputs::  When you get values from one command.


File: gnuastro.info,  Node: Separate shell variables for multiple outputs,  Prev: Shell tips,  Up: Shell tips

4.1.5.1 Separate shell variables for multiple outputs
.....................................................

Sometimes your commands print multiple values and you want to use them
as different shell variables.  Let’s describe the problem (shown in the
box below) with an example (that you can reproduce without any external
data).

   With the commands below, we’ll first make a noisy ($\sigma=5$) image
($100\times100$ pixels) using *note Arithmetic::.  Then, we’ll
measure(1) its mean and standard deviation using *note Statistics::.

     $ astarithmetic 100 100 2 makenew 5 mknoise-sigma -oimg.fits

     $ aststatistics img.fits --mean --std
     -3.10938611484039e-03 4.99607077069093e+00

*THE PROBLEM:* you want the first number printed above to be stored in a
shell variable called ‘my_mean’ and the second number to be stored as
the ‘my_std’ shell variable (you are free to choose any name!).

The first thing that may come to mind is to run Statistics two times,
and write the output into separate variables like below:

     $ my_std=$(aststatistics img.fits --std)           ## NOT SOLUTION! ##
     $ my_mean=$(aststatistics img.fits --mean)         ## NOT SOLUTION! ##

   But this is not a good solution because as ‘img.fits’ becomes larger
(more pixels), the time it takes for Statistics to simply load the data
into memory can be significant.  This will slow down your pipeline and
besides wasting your time, it contributes to global warming (by spending
energy on an un-necessary action; take this seriously because your
pipeline may scale up to involve thousands of large datasets)!
Furthermore, besides loading of the input data, Statistics (and Gnuastro
in general) is designed to do multiple measurements in one pass over the
data as much as possible (to further decrease Gnuastro’s carbon
footprint).  So when given ‘--mean --std’, it will measure both in one
pass over the pixels (not two passes!).  In other words, in this case,
you get the two measurements for the cost of one.

   How do you separate the values from the first ‘aststatistics’ command
above?  One ugly way is to write the two-number output string into a
single shell variable and then separate, or tokenize, the string with
two subsequent commands like below:

     $ meanstd=$(aststatistics img.fits --mean --std)   ## NOT SOLUTION! ##
     $ my_mean=$(echo $meanstd | awk '{print $1}')      ## NOT SOLUTION! ##
     $ my_std=$(echo $meanstd | awk '{print $2}')       ## NOT SOLUTION! ##

*SOLUTION:* The solution is to formatted-print (‘printf’) the numbers as
shell variables definitions in a string, and evaluate (‘eval’) that
string as a command:

     $ eval "$(aststatistics img.fits --mean --std \
                             | xargs printf "my_mean=%s; my_std=%s")"

Let’s review the solution (in more detail):

  1. We pipe the output into ‘xargs’(2) (extended arguments) which puts
     the two numbers it gets from the pipe, as arguments for ‘printf’
     (formatted print; because ‘printf’ doesn’t take input from pipes).
  2. Within the ‘printf’ call, we write the values after putting a
     variable name and equal-sign, and in between them we put a <;> (as
     if it was a shell command).  The ‘%s’ tells ‘printf’ to print each
     input as a string (not to interpret it as a number and loose
     precision).  Here is the output of this phase:

          $ aststatistics img.fits --mean --std \
                                | xargs printf "my_mean=%s; my_std=%s"
          my_mean=-3.10938611484039e-03; my_std=4.99607077069093e+00

  3. But the output above is a string!  To evaluate this string as a
     command, we give it to the eval command like above.

After the solution above, you will have the two ‘my_mean’ and ‘my_std’
variables to use separately in your pipeline:

     $ echo $my_mean
     -3.10938611484039e-03
     $ echo $my_std
     4.99607077069093e+00

   This ‘eval’-based solution has been tested in in GNU Bash, Dash and
Zsh and it works nicely in them (is “portable”).  This is because the
constructs used here are pretty low-level (and widely available).

   For examples usages of this technique, see the following sections:
*note Extracting a single spectrum and plotting it:: and *note Pseudo
narrow-band images::.

   ---------- Footnotes ----------

   (1) The actual printed values by ‘aststatistics’ may slightly differ
for you.  This is because of a different random number generator seed
used in ‘astarithmetic’.  To get an exactly reproducible result, see
*note Generating random numbers::

   (2) For more on ‘xargs’, see <https://en.wikipedia.org/wiki/Xargs>.
It will take the standard input (from the pipe in this scenario) and put
it as arguments of the next program (‘printf’ in this scenario).  In
other words, it is good for programs that don’t take input from standard
input (‘printf’ in this case; but also includes others like ‘cp’, ‘rm’,
or ‘echo’).


File: gnuastro.info,  Node: Configuration files,  Next: Getting help,  Prev: Command-line,  Up: Common program behavior

4.2 Configuration files
=======================

Each program needs a certain number of parameters to run.  Supplying all
the necessary parameters each time you run the program is very
frustrating and prone to errors.  Therefore all the programs read the
values for the necessary options you have not given in the command-line
from one of several plain text files (which you can view and edit with
any text editor).  These files are known as configuration files and are
usually kept in a directory named ‘etc/’ according to the file system
hierarchy standard(1).

   The thing to have in mind is that none of the programs in Gnuastro
keep any internal default value.  All the values must either be stored
in one of the configuration files or explicitly called in the
command-line.  In case the necessary parameters are not given through
any of these methods, the program will print a missing option error and
abort.  The only exception to this is ‘--numthreads’, whose default
value is determined at run-time using the number of threads available to
your system, see *note Multi-threaded operations::.  Of course, you can
still provide a default value for the number of threads at any of the
levels below, but if you do not, the program will not abort.  Also note
that through automatic output name generation, the value to the
‘--output’ option is also not mandatory on the command-line or in the
configuration files for all programs which do not rely on that value as
an input(2), see *note Automatic output::.

* Menu:

* Configuration file format::   ASCII format of configuration file.
* Configuration file precedence::  Precedence of configuration files.
* Current directory and User wide::  Local and user configuration files.
* System wide::                 System wide configuration files.

   ---------- Footnotes ----------

   (1) <http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard>

   (2) One example of a program which uses the value given to ‘--output’
as an input is ConvertType, this value specifies the type of the output
through the value to ‘--output’, see *note Invoking astconvertt::.


File: gnuastro.info,  Node: Configuration file format,  Next: Configuration file precedence,  Prev: Configuration files,  Up: Configuration files

4.2.1 Configuration file format
-------------------------------

The configuration files for each program have the standard program
executable name with a ‘‘.conf’’ suffix.  When you download the source
code, you can find them in the same directory as the source code of each
program, see *note Program source::.

   Any line in the configuration file whose first non-white character is
a <#> is considered to be a comment and is ignored.  An empty line is
also similarly ignored.  The long name of the option should be used as
an identifier.  The option name and option value should be separated by
any number of ‘white-space’ characters (space, tab or vertical tab) or
an equal (<=>).  By default several space characters are used.  If the
value of an option has space characters (most commonly for the ‘hdu’
option), then the full value can be enclosed in double quotation signs
(<">, similar to the example in *note Arguments and options::).  If it
is an option without a value in the ‘--help’ output (on/off option, see
*note Options::), then the value should be ‘1’ if it is to be ‘on’ and
‘0’ otherwise.

   In each non-commented and non-blank line, any text after the first
two words (option identifier and value) is ignored.  If an option
identifier is not recognized in the configuration file, the name of the
file, the line number of the unrecognized option, and the unrecognized
identifier name will be reported and the program will abort.  If a
parameter is repeated more more than once in the configuration files,
accepts only one value, and is not set on the command-line, then only
the first value will be used, the rest will be ignored.

   You can build or edit any of the directories and the configuration
files yourself using any text editor.  However, it is recommended to use
the ‘--setdirconf’ and ‘--setusrconf’ options to set default values for
the current directory or this user, see *note Operating mode options::.
With these options, the values you give will be checked before writing
in the configuration file.  They will also print a set of commented
lines guiding the reader and will also classify the options based on
their context and write them in their logical order to be more
understandable.


File: gnuastro.info,  Node: Configuration file precedence,  Next: Current directory and User wide,  Prev: Configuration file format,  Up: Configuration files

4.2.2 Configuration file precedence
-----------------------------------

The option values in all the programs of Gnuastro will be filled in the
following order.  If an option only takes one value which is given in an
earlier step, any value for that option in a later step will be ignored.
Note that if the ‘lastconfig’ option is specified in any step below, no
other configuration files will be parsed (see *note Operating mode
options::).

  1. Command-line options, for a particular run of ProgramName.

  2. ‘.gnuastro/astprogname.conf’ is parsed by ProgramName in the
     current directory.

  3. ‘.gnuastro/gnuastro.conf’ is parsed by all Gnuastro programs in the
     current directory.

  4. ‘$HOME/.local/etc/astprogname.conf’ is parsed by ProgramName in the
     user’s home directory (see *note Current directory and User
     wide::).

  5. ‘$HOME/.local/etc/gnuastro.conf’ is parsed by all Gnuastro programs
     in the user’s home directory (see *note Current directory and User
     wide::).

  6. ‘prefix/etc/astprogname.conf’ is parsed by ProgramName in the
     system-wide installation directory (see *note System wide:: for
     ‘prefix’).

  7. ‘prefix/etc/gnuastro.conf’ is parsed by all Gnuastro programs in
     the system-wide installation directory (see *note System wide:: for
     ‘prefix’).

   The basic idea behind setting this progressive state of checking for
parameter values is that separate users of a computer or separate
folders in a user’s file system might need different values for some
parameters.

*Checking the order:* You can confirm/check the order of parsing
configuration files using the ‘--checkconfig’ option with any Gnuastro
program, see *note Operating mode options::.  Just be sure to place this
option immediately after the program name, before any other option.

   As you see above, there can also be a configuration file containing
the common options in all the programs: ‘gnuastro.conf’ (see *note
Common options::).  If options specific to one program are specified in
this file, there will be unrecognized option errors, or unexpected
behavior if the option has different behavior in another program.  On
the other hand, there is no problem with ‘astprogname.conf’ containing
common options(1).

*Manipulating the order:* You can manipulate this order or add new files
with the following two options which are fully described in *note
Operating mode options:::
‘--config’
     Allows you to define any file to be parsed as a configuration file
     on the command-line or within the any other configuration file.
     Recall that the file given to ‘--config’ is parsed immediately when
     this option is confronted (on the command-line or in a
     configuration file).

‘--lastconfig’
     Allows you to stop the parsing of subsequent configuration files.
     Note that if this option is given in a configuration file, it will
     be fully read, so its position in the configuration does not matter
     (unlike ‘--config’).

   One example of benefiting from these configuration files can be this:
raw telescope images usually have their main image extension in the
second FITS extension, while processed FITS images usually only have one
extension.  If your system-wide default input extension is 0 (the
first), then when you want to work with the former group of data you
have to explicitly mention it to the programs every time.  With this
progressive state of default values to check, you can set different
default values for the different directories that you would like to run
Gnuastro in for your different purposes, so you will not have to worry
about this issue any more.

   The same can be said about the ‘gnuastro.conf’ files: by specifying a
behavior in this single file, all Gnuastro programs in the respective
directory, user, or system-wide steps will behave similarly.  For
example, to keep the input’s directory when no specific output is given
(see *note Automatic output::), or to not delete an existing file if it
has the same name as a given output (see *note Input output options::).

   ---------- Footnotes ----------

   (1) As an example, the ‘--setdirconf’ and ‘--setusrconf’ options will
also write the common options they have read in their produced
‘astprogname.conf’.


File: gnuastro.info,  Node: Current directory and User wide,  Next: System wide,  Prev: Configuration file precedence,  Up: Configuration files

4.2.3 Current directory and User wide
-------------------------------------

For the current (local) and user-wide directories, the configuration
files are stored in the hidden sub-directories named ‘.gnuastro/’ and
‘$HOME/.local/etc/’ respectively.  Unless you have changed it, the
‘$HOME’ environment variable should point to your home directory.  You
can check it by running ‘$ echo $HOME’.  Each time you run any of the
programs in Gnuastro, this environment variable is read and placed in
the above address.  So if you suddenly see that your home configuration
files are not being read, probably you (or some other program) has
changed the value of this environment variable.

   Although it might cause confusions like above, this dependence on the
‘HOME’ environment variable enables you to temporarily use a different
directory as your home directory.  This can come in handy in complicated
situations.  To set the user or current directory configuration files
based on your command-line input, you can use the ‘--setdirconf’ or
‘--setusrconf’, see *note Operating mode options::.


File: gnuastro.info,  Node: System wide,  Prev: Current directory and User wide,  Up: Configuration files

4.2.4 System wide
-----------------

When Gnuastro is installed, the configuration files that are shipped
with the distribution are copied into the (possibly system wide)
‘prefix/etc/’ directory.  For more details on ‘prefix’, see *note
Installation directory:: (by default it is: ‘/usr/local’).  This
directory is the final place (with the lowest priority) that the
programs in Gnuastro will check to retrieve parameter values.

   If you remove an option and its value from the system wide
configuration files, you either have to specify it in more immediate
configuration files or set it each time in the command-line.  Recall
that none of the programs in Gnuastro keep any internal default values
and will abort if they do not find a value for the necessary parameters
(except the number of threads and output file name).  So even though you
might never expect to use an optional option, it safe to have it
available in this system-wide configuration file even if you do not
intend to use it frequently.

   Note that in case you install Gnuastro from your distribution’s
repositories, ‘prefix’ will either be set to ‘/’ (the root directory) or
‘/usr’, so you can find the system wide configuration variables in
‘/etc/’ or ‘/usr/etc/’.  The prefix of ‘/usr/local/’ is conventionally
used for programs you install from source by yourself as in *note Quick
start::.


File: gnuastro.info,  Node: Getting help,  Next: Multi-threaded operations,  Prev: Configuration files,  Up: Common program behavior

4.3 Getting help
================

Probably the first time you read this book, it is either in the PDF or
HTML formats.  These two formats are very convenient for when you are
not actually working, but when you are only reading.  Later on, when you
start to use the programs and you are deep in the middle of your work,
some of the details will inevitably be forgotten.  Going to find the PDF
file (printed or digital) or the HTML web page is a major distraction.

   GNU software have a very unique set of tools for aiding your memory
on the command-line, where you are working, depending how much of it you
need to remember.  In the past, such command-line help was known as
“online” help, because they were literally provided to you ‘on’ the
command ‘line’.  However, nowadays the word “online” refers to something
on the internet, so that term will not be used.  With this type of help,
you can resume your exciting research without taking your hands off the
keyboard.

   Another major advantage of such command-line based help routines is
that they are installed with the software in your computer, therefore
they are always in sync with the executable you are actually running.
Three of them are actually part of the executable.  You do not have to
worry about the version of the book or program.  If you rely on external
help (a PDF in your personal print or digital archive or HTML from the
official web page) you have to check to see if their versions fit with
your installed program.

   If you only need to remember the short or long names of the options,
‘--usage’ is advised.  If it is what the options do, then ‘--help’ is a
great tool.  Man pages are also provided for those who are use to this
older system of documentation.  This full book is also available to you
on the command-line in Info format.  If none of these seems to resolve
the problems, there is a mailing list which enables you to get in touch
with experienced Gnuastro users.  In the subsections below each of these
methods are reviewed.

* Menu:

* --usage::                     View option names and value formats.
* --help::                      List all options with description.
* Man pages::                   Man pages generated from –help.
* Info::                        View complete book in terminal.
* help-gnuastro mailing list::  Contacting experienced users.


File: gnuastro.info,  Node: --usage,  Next: --help,  Prev: Getting help,  Up: Getting help

4.3.1 ‘--usage’
---------------

If you give this option, the program will not run.  It will only print a
very concise message showing the options and arguments.  Everything
within square brackets (‘[]’) is optional.  For example, here are the
first and last two lines of Crop’s ‘--usage’ is shown:

     $ astcrop --usage
     Usage: astcrop [-Do?IPqSVW] [-d INT] [-h INT] [-r INT] [-w INT]
                 [-x INT] [-y INT] [-c INT] [-p STR] [-N INT] [--deccol=INT]
                 ....
                 [--setusrconf] [--usage] [--version] [--wcsmode]
                 [ASCIIcatalog] FITSimage(s).fits

   There are no explanations on the options, just their short and long
names shown separately.  After the program name, the short format of all
the options that do not require a value (on/off options) is displayed.
Those that do require a value then follow in separate brackets, each
displaying the format of the input they want, see *note Options::.
Since all options are optional, they are shown in square brackets, but
arguments can also be optional.  For example, in this example, a catalog
name is optional and is only required in some modes.  This is a standard
method of displaying optional arguments for all GNU software.


File: gnuastro.info,  Node: --help,  Next: Man pages,  Prev: --usage,  Up: Getting help

4.3.2 ‘--help’
--------------

If the command-line includes this option, the program will not be run.
It will print a complete list of all available options along with a
short explanation.  The options are also grouped by their context.
Within each context, the options are sorted alphabetically.  Since the
options are shown in detail afterwards, the first line of the ‘--help’
output shows the arguments and if they are optional or not, similar to
*note --usage::.

   In the ‘--help’ output of all programs in Gnuastro, the options for
each program are classified based on context.  The first two contexts
are always options to do with the input and output respectively.  For
example, input image extensions or supplementary input files for the
inputs.  The last class of options is also fixed in all of Gnuastro, it
shows operating mode options.  Most of these options are already
explained in *note Operating mode options::.

   The help message will sometimes be longer than the vertical size of
your terminal.  If you are using a graphical user interface terminal
emulator, you can scroll the terminal with your mouse, but we promised
no mice distractions!  So here are some suggestions:

   • <Shift + PageUP> to scroll up and <Shift + PageDown> to scroll
     down.  For most help output this should be enough.  The problem is
     that it is limited by the number of lines that your terminal keeps
     in memory and that you cannot scroll by lines, only by whole
     screens.

   • Pipe to ‘less’.  A pipe is a form of shell re-direction.  The
     ‘less’ tool in Unix-like systems was made exactly for such outputs
     of any length.  You can pipe (‘|’) the output of any program that
     is longer than the screen to it and then you can scroll through (up
     and down) with its many tools.  For example:
          $ astnoisechisel --help | less
     Once you have gone through the text, you can quit ‘less’ by
     pressing the <q> key.

   • Redirect to a file.  This is a less convenient way, because you
     will then have to open the file in a text editor!  You can do this
     with the shell redirection tool (‘>’):
          $ astnoisechisel --help > filename.txt

   In case you have a special keyword you are looking for in the help,
you do not have to go through the full list.  GNU Grep is made for this
job.  For example, if you only want the list of options whose ‘--help’
output contains the word “axis” in Crop, you can run the following
command:

     $ astcrop --help | grep axis

   If the output of this option does not fit nicely within the confines
of your terminal, GNU does enable you to customize its output through
the environment variable ‘ARGP_HELP_FMT’, you can set various parameters
which specify the formatting of the help messages.  For example, if your
terminals are wider than 70 spaces (say 100) and you feel there is too
much empty space between the long options and the short explanation, you
can change these formats by giving values to this environment variable
before running the program with the ‘--help’ output.  You can define
this environment variable in this manner:
     $ export ARGP_HELP_FMT=rmargin=100,opt-doc-col=20
   This will affect all GNU programs using GNU C library’s ‘argp.h’
facilities as long as the environment variable is in memory.  You can
see the full list of these formatting parameters in the “Argp User
Customization” part of the GNU C library manual.  If you are more
comfortable to read the ‘--help’ outputs of all GNU software in your
customized format, you can add your customization (similar to the line
above, without the ‘$’ sign) to your ‘~/.bashrc’ file.  This is a
standard option for all GNU software.


File: gnuastro.info,  Node: Man pages,  Next: Info,  Prev: --help,  Up: Getting help

4.3.3 Man pages
---------------

Man pages were the Unix method of providing command-line documentation
to a program.  With GNU Info, see *note Info:: the usage of this method
of documentation is highly discouraged.  This is because Info provides a
much more easier to navigate and read environment.

   However, some operating systems require a man page for packages that
are installed and some people are still used to this method of
command-line help.  So the programs in Gnuastro also have Man pages
which are automatically generated from the outputs of ‘--version’ and
‘--help’ using the GNU help2man program.  So if you run
     $ man programname
You will be provided with a man page listing the options in the standard
manner.


File: gnuastro.info,  Node: Info,  Next: help-gnuastro mailing list,  Prev: Man pages,  Up: Getting help

4.3.4 Info
----------

Info is the standard documentation format for all GNU software.  It is a
very useful command-line document viewing format, fully equipped with
links between the various pages and menus and search capabilities.  As
explained before, the best thing about it is that it is available for
you the moment you need to refresh your memory on any command-line tool
in the middle of your work without having to take your hands off the
keyboard.  This complete book is available in Info format and can be
accessed from anywhere on the command-line.

   To open the Info format of any installed programs or library on your
system which has an Info format book, you can simply run the command
below (change ‘executablename’ to the executable name of the program or
library):

     $ info executablename

In case you are not already familiar with it, run ‘$ info info’.  It
does a fantastic job in explaining all its capabilities itself.  It is
very short and you will become sufficiently fluent in about half an
hour.  Since all GNU software documentation is also provided in Info,
your whole GNU/Linux life will significantly improve.

   Once you’ve become an efficient navigator in Info, you can go to any
part of this book or any other GNU software or library manual, no matter
how long it is, in a matter of seconds.  It also blends nicely with GNU
Emacs (a text editor) and you can search manuals while you are writing
your document or programs without taking your hands off the keyboard,
this is most useful for libraries like the GNU C library.  To be able to
access all the Info manuals installed in your GNU/Linux within Emacs,
type <Ctrl-H + i>.

   To see this whole book from the beginning in Info, you can run

     $ info gnuastro

If you run Info with the particular program executable name, for example
‘astcrop’ or ‘astnoisechisel’:

     $ info astprogramname

you will be taken to the section titled “Invoking ProgramName” which
explains the inputs and outputs along with the command-line options for
that program.  Finally, if you run Info with the official program name,
for example, Crop or NoiseChisel:

     $ info ProgramName

you will be taken to the top section which introduces the program.  Note
that in all cases, Info is not case sensitive.


File: gnuastro.info,  Node: help-gnuastro mailing list,  Prev: Info,  Up: Getting help

4.3.5 help-gnuastro mailing list
--------------------------------

Gnuastro maintains the help-gnuastro mailing list for users to ask any
questions related to Gnuastro.  The experienced Gnuastro users and some
of its developers are subscribed to this mailing list and your email
will be sent to them immediately.  However, when contacting this mailing
list please have in mind that they are possibly very busy and might not
be able to answer immediately.

   To ask a question from this mailing list, send a mail to
‘help-gnuastro@gnu.org’.  Anyone can view the mailing list archives at
<http://lists.gnu.org/archive/html/help-gnuastro/>.  It is best that
before sending a mail, you search the archives to see if anyone has
asked a question similar to yours.  If you want to make a suggestion or
report a bug, please do not send a mail to this mailing list.  We have
other mailing lists and tools for those purposes, see *note Report a
bug:: or *note Suggest new feature::.


File: gnuastro.info,  Node: Multi-threaded operations,  Next: Numeric data types,  Prev: Getting help,  Up: Common program behavior

4.4 Multi-threaded operations
=============================

Some of the programs benefit significantly when you use all the threads
your computer’s CPU has to offer to your operating system.  The number
of threads available can be larger than the number of physical
(hardware) cores in the CPU (also known as Simultaneous multithreading).
For example, in Intel’s CPUs (those that implement its Hyper-threading
technology) the number of threads is usually double the number of
physical cores in your CPU. On a GNU/Linux system, the number of threads
available can be found with the command ‘$ nproc’ command (part of GNU
Coreutils).

   Gnuastro’s programs can find the number of threads available to your
system internally at run-time (when you execute the program).  However,
if a value is given to the ‘--numthreads’ option, the given number will
be used, see *note Operating mode options:: and *note Configuration
files:: for ways to use this option.  Thus ‘--numthreads’ is the only
common option in Gnuastro’s programs with a value that does not have to
be specified anywhere on the command-line or in the configuration files.

* Menu:

* A note on threads::           Caution and suggestion on using threads.
* How to run simultaneous operations::  How to run things simultaneously.


File: gnuastro.info,  Node: A note on threads,  Next: How to run simultaneous operations,  Prev: Multi-threaded operations,  Up: Multi-threaded operations

4.4.1 A note on threads
-----------------------

Spinning off threads is not necessarily the most efficient way to run an
application.  Creating a new thread is not a cheap operation for the
operating system.  It is most useful when the input data are fixed and
you want the same operation to be done on parts of it.  For example, one
input image to Crop and multiple crops from various parts of it.  In
this fashion, the image is loaded into memory once, all the crops are
divided between the number of threads internally and each thread cuts
out those parts which are assigned to it from the same image.  On the
other hand, if you have multiple images and you want to crop the same
region(s) out of all of them, it is much more efficient to set
‘--numthreads=1’ (so no threads spin off) and run Crop multiple times
simultaneously, see *note How to run simultaneous operations::.

   You can check the boost in speed by first running a program on one of
the data sets with the maximum number of threads and another time (with
everything else the same) and only using one thread.  You will notice
that the wall-clock time (reported by most programs at their end) in the
former is longer than the latter divided by number of physical CPU cores
(not threads) available to your operating system.  Asymptotically these
two times can be equal (most of the time they are not).  So limiting the
programs to use only one thread and running them independently on the
number of available threads will be more efficient.

   Note that the operating system keeps a cache of recently processed
data, so usually, the second time you process an identical data set
(independent of the number of threads used), you will get faster
results.  In order to make an unbiased comparison, you have to first
clean the system’s cache with the following command between the two
runs.

     $ sync; echo 3 | sudo tee /proc/sys/vm/drop_caches

*SUMMARY: Should I use multiple threads?*  Depends:
   • If you only have *one* data set (image in most cases!), then yes,
     the more threads you use (with a maximum of the number of threads
     available to your OS) the faster you will get your results.

   • If you want to run the same operation on *multiple* data sets, it
     is best to set the number of threads to 1 and use Make, or GNU
     Parallel, as explained in *note How to run simultaneous
     operations::.


File: gnuastro.info,  Node: How to run simultaneous operations,  Prev: A note on threads,  Up: Multi-threaded operations

4.4.2 How to run simultaneous operations
----------------------------------------

There are two(1) approaches to simultaneously execute a program: using
GNU Parallel or Make (GNU Make is the most common implementation).  The
first is very useful when you only want to do one job multiple times and
want to get back to your work without actually keeping the command you
ran.  The second is usually for more important operations, with lots of
dependencies between the different products (for example, a full
scientific research).

GNU Parallel
     When you only want to run multiple instances of a command on
     different threads and get on with the rest of your work, the best
     method is to use GNU parallel.  Surprisingly GNU Parallel is one of
     the few GNU packages that has no Info documentation but only a Man
     page, see *note Info::.  So to see the documentation after
     installing it please run

          $ man parallel
     As an example, let’s assume we want to crop a region fixed on the
     pixels (500, 600) with the default width from all the FITS images
     in the ‘./data’ directory ending with ‘sci.fits’ to the current
     directory.  To do this, you can run:

          $ parallel astcrop --numthreads=1 --xc=500 --yc=600 ::: \
            ./data/*sci.fits

     GNU Parallel can help in many more conditions, this is one of the
     simplest, see the man page for lots of other examples.  For
     absolute beginners: the backslash (‘\’) is only a line breaker to
     fit nicely in the page.  If you type the whole command in one line,
     you should remove it.

Make
     Make is a program for building “targets” (e.g., files) using
     “recipes” (a set of operations) when their known “prerequisites”
     (other files) have been updated.  It elegantly allows you to define
     dependency structures for building your final output and updating
     it efficiently when the inputs change.  It is the most common
     infra-structure to build software today.

     Scientific research methodology is very similar to software
     development: you start by testing a hypothesis on a small sample of
     objects/targets with a simple set of steps.  As you are able to get
     promising results, you improve the method and use it on a larger,
     more general, sample.  In the process, you will confront many
     issues that have to be corrected (bugs in software development
     jargon).  Make is a wonderful tool to manage this style of
     development.

     Besides the raw data analysis pipeline, Make has been used to for
     producing reproducible papers, for example, see the reproduction
     pipeline (https://gitlab.com/makhlaghi/NoiseChisel-paper) of the
     paper introducing *note NoiseChisel:: (one of Gnuastro’s programs).
     In fact the NoiseChisel paper’s Make-based workflow was the
     foundation of a parallel project called Maneage
     (http://maneage.org) (_Man_aging data lin_eage_):
     <http://maneage.org> that is described more fully in Akhlaghi et
     al.  2021 (https://arxiv.org/abs/2006.03018).  Therefore, it is a
     very useful tool for complex scientific workflows.

     GNU Make(2) is the most common implementation which (similar to
     nearly all GNU programs, comes with a wonderful manual(3)).  Make
     is very basic and simple, and thus the manual is short (the most
     important parts are in the first roughly 100 pages) and easy to
     read/understand.

     Make comes with a ‘--jobs’ (‘-j’) option which allows you to
     specify the maximum number of jobs that can be done simultaneously.
     For example, if you have 8 threads available to your operating
     system.  You can run:

          $ make -j8

     With this command, Make will process your ‘Makefile’ and create all
     the targets (can be thousands of FITS images for example)
     simultaneously on 8 threads, while fully respecting their
     dependencies (only building a file/target when its prerequisites
     are successfully built).  Make is thus strongly recommended for
     managing scientific research where robustness, archiving,
     reproducibility and speed(4) are important.

   ---------- Footnotes ----------

   (1) A third way would be to open multiple terminal emulator windows
in your GUI, type the commands separately on each and press <Enter> once
on each terminal, but this is far too frustrating, tedious and prone to
errors.  It’s therefore not a realistic solution when tens, hundreds or
thousands of operations (your research targets, multiplied by the
operations you do on each) are to be done.

   (2) <https://www.gnu.org/software/make/>

   (3) <https://www.gnu.org/software/make/manual/>

   (4) Besides its multi-threaded capabilities, Make will only re-build
those targets that depend on a change you have made, not the whole work.
For example, if you have set the prerequisites properly, you can easily
test the changing of a parameter on your paper’s results without having
to re-do everything (which is much faster).  This allows you to be much
more productive in easily checking various ideas/assumptions of the
different stages of your research and thus produce a more robust result
for your exciting science.


File: gnuastro.info,  Node: Numeric data types,  Next: Memory management,  Prev: Multi-threaded operations,  Up: Common program behavior

4.5 Numeric data types
======================

At the lowest level, the computer stores everything in terms of ‘1’ or
‘0’.  For example, each program in Gnuastro, or each astronomical image
you take with the telescope is actually a string of millions of these
zeros and ones.  The space required to keep a zero or one is the
smallest unit of storage, and is known as a _bit_.  However,
understanding and manipulating this string of bits is extremely hard for
most people.  Therefore, different standards are defined to package the
bits into separate _type_s with a fixed interpretation of the bits in
each package.

   To store numbers, the most basic standard/type is for integers ($...,
-2, -1, 0, 1, 2, ...$).  The common integer types are 8, 16, 32, and 64
bits wide (more bits will give larger limits).  Each bit corresponds to
a power of 2 and they are summed to create the final number.  In the
integer types, for each width there are two standards for reading the
bits: signed and unsigned.  In the ‘signed’ convention, one bit is
reserved for the sign (stating that the integer is positive or
negative).  The ‘unsigned’ integers use that bit in the actual number
and thus contain only positive numbers (starting from zero).

   Therefore, at the same number of bits, both signed and unsigned
integers can allow the same number of integers, but the positive limit
of the ‘unsigned’ types is double their ‘signed’ counterparts with the
same width (at the expense of not having negative numbers).  When the
context of your work does not involve negative numbers (for example,
counting, where negative is not defined), it is best to use the
‘unsigned’ types.  For the full numerical range of all integer types,
see below.

   Another standard of converting a given number of bits to numbers is
the floating point standard, this standard can _approximately_ store any
real number with a given precision.  There are two common floating point
types: 32-bit and 64-bit, for single and double precision floating point
numbers respectively.  The former is sufficient for data with less than
8 significant decimal digits (most astronomical data), while the latter
is good for less than 16 significant decimal digits.  The representation
of real numbers as bits is much more complex than integers.  If you are
interested to learn more about it, you can start with the Wikipedia
article (https://en.wikipedia.org/wiki/Floating_point).

   Practically, you can use Gnuastro’s Arithmetic program to
convert/change the type of an image/datacube (see *note Arithmetic::),
or Gnuastro Table program to convert a table column’s data type (see
*note Column arithmetic::).  Conversion of a dataset’s type is necessary
in some contexts.  For example, the program/library, that you intend to
feed the data into, only accepts floating point values, but you have an
integer image/column.  Another situation that conversion can be helpful
is when you know that your data only has values that fit within ‘int8’
or ‘uint16’.  However it is currently formatted in the ‘float64’ type.

   The important thing to consider is that operations involving wider,
floating point, or signed types can be significantly slower than
smaller-width, integer, or unsigned types respectively.  Note that
besides speed, a wider type also requires much more storage space (by 4
or 8 times).  Therefore, when you confront such situations that can be
optimized and want to store/archive/transfer the data, it is best to use
the most efficient type.  For example, if your dataset (image or table
column) only has positive integers less than 65535, store it as an
unsigned 16-bit integer for faster processing, faster transfer, and less
storage space.

   The short and long names for the recognized numeric data types in
Gnuastro are listed below.  Both short and long names can be used when
you want to specify a type.  For example, as a value to the common
option ‘--type’ (see *note Input output options::), or in the
information comment lines of *note Gnuastro text table format::.  The
ranges listed below are inclusive.

‘u8’
‘uint8’
     8-bit unsigned integers, range:
     $[0\rm{\ to\ }2^8-1]$ or $[0\rm{\ to\ }255]$.

‘i8’
‘int8’
     8-bit signed integers, range:
     $[-2^7\rm{\ to\ }2^7-1]$ or $[-128\rm{\ to\ }127]$.

‘u16’
‘uint16’
     16-bit unsigned integers, range:
     $[0\rm{\ to\ }2^{16}-1]$ or $[0\rm{\ to\ }65535]$.

‘i16’
‘int16’
     16-bit signed integers, range:
     $[-2^{15}\rm{\ to\ }2^{15}-1]$ or $[-32768\rm{\ to\ }32767]$.

‘u32’
‘uint32’
     32-bit unsigned integers, range:
     $[0\rm{\ to\ }2^{32}-1]$ or $[0\rm{\ to\ }4294967295]$.

‘i32’
‘int32’
     32-bit signed integers, range:
     $[-2^{31}\rm{\ to\ }2^{31}-1]$ or $[-2147483648\rm{\ to\
     }2147483647]$.

‘u64’
‘uint64’
     64-bit unsigned integers, range
     $[0\rm{\ to\ }2^{64}-1]$ or $[0\rm{\ to\ }18446744073709551615]$.

‘i64’
‘int64’
     64-bit signed integers, range:
     $[-2^{63}\rm{\ to\ }2^{63}-1]$ or $[-9223372036854775808\rm{\ to\
     }9223372036854775807]$.

‘f32’
‘float32’
     32-bit (single-precision) floating point types.  The maximum
     (minimum is its negative) possible value is
     $3.402823\times10^{38}$.  Single-precision floating points can
     accurately represent a floating point number up to $\sim7.2$
     significant decimals.  Given the heavy noise in astronomical data,
     this is usually more than sufficient for storing results.  For
     more, see *note Printing floating point numbers::.

‘f64’
‘float64’
     64-bit (double-precision) floating point types.  The maximum
     (minimum is its negative) possible value is $\sim10^{308}$.
     Double-precision floating points can accurately represent a
     floating point number $\sim15.9$ significant decimals.  This is
     usually good for processing (mixing) the data internally, for
     example, a sum of single precision data (and later storing the
     result as ‘float32’).  For more, see *note Printing floating point
     numbers::.

*Some file formats do not recognize all types.*  for example, the FITS
standard (see *note Fits::) does not define ‘uint64’ in binary tables or
images.  When a type is not acceptable for output into a given file
format, the respective Gnuastro program or library will let you know and
abort.  On the command-line, you can convert the numerical type of an
image, or table column into another type with *note Arithmetic:: or
*note Table:: respectively.  If you are writing your own program, you
can use the ‘gal_data_copy_to_new_type()’ function in Gnuastro’s
library, see *note Copying datasets::.


File: gnuastro.info,  Node: Memory management,  Next: Tables,  Prev: Numeric data types,  Up: Common program behavior

4.6 Memory management
=====================

In this section we will review how Gnuastro manages your input data in
your system’s memory.  Knowing this can help you optimize your usage (in
speed and memory consumption) when the data volume is large and
approaches, or exceeds, your available RAM (usually in various calls to
multiple programs simultaneously).  But before diving into the details,
let’s have a short basic introduction to memory in general and in
particular the types of memory most relevant to this discussion.

   Input datasets (that are later fed into programs for analysis) are
commonly first stored in _non-volatile memory_.  This is a type of
memory that does not need a constant power supply to keep the data and
is therefore primarily aimed for long-term storage, like HDDs or SSDs.
So data in this type of storage is preserved when you turn off your
computer.  But by its nature, non-volatile memory is much slower, in
reading or writing, than the speeds that CPUs can process the data.
Thus relying on this type of memory alone would create a bad bottleneck
in the input/output (I/O) phase of any processing.

   The first step to decrease this bottleneck is to have a faster
storage space, but with a much limited storage volume.  For this type of
storage, computers have a Random Access Memory (or RAM). RAM is
classified as a _volatile memory_ because it needs a constant flow of
electricity to keep the information.  In other words, the moment power
is cut-off, all the stored information in your RAM is gone (hence the
“volatile” name).  But thanks to that constant supply of power, it can
access any random address with equal (and very high!)  speed.

   Hence, the general/simplistic way that programs deal with memory is
the following (this is general to almost all programs, not just
Gnuastro’s): 1) Load/copy the input data from the non-volatile memory
into RAM. 2) Use the copy of the data in RAM as input for all the
internal processing as well as the intermediate data that is necessary
during the processing.  3) Finally, when the analysis is complete, write
the final output data back into non-volatile memory, and free/delete all
the used space in the RAM (the initial copy and all the intermediate
data).  Usually the RAM is most important for the data of the
intermediate steps (that you never see as a user of a program!).

   When the input dataset(s) to a program are small (compared to the
available space in your system’s RAM at the moment it is run) Gnuastro’s
programs and libraries follow the standard series of steps above.  The
only exception is that deleting the intermediate data is not only done
at the end of the program.  As soon as an intermediate dataset is no
longer necessary for the next internal steps, the space it occupied is
deleted/freed.  This allows Gnuastro programs to minimize their usage of
your system’s RAM over the full running time.

   The situation gets complicated when the datasets are large (compared
to your available RAM when the program is run).  For example, if a
dataset is half the size of your system’s available RAM, and the
program’s internal analysis needs three or more intermediately processed
copies of it at one moment in its analysis.  There will not be enough
RAM to keep those higher-level intermediate data.  In such cases,
programs that do not do any memory management will crash.  But
fortunately Gnuastro’s programs do have a memory management plans for
such situations.

   When the necessary amount of space for an intermediate dataset cannot
be allocated in the RAM, Gnuastro’s programs will not use the RAM at
all.  They will use the “memory-mapped file” concept in modern operating
systems to create a randomly-named file in your non-volatile memory and
use that instead of the RAM. That file will have the exact size (in
bytes) of that intermediate dataset.  Any time the program needs that
intermediate dataset, the operating system will directly go to that
file, and bypass your RAM. As soon as that file is no longer necessary
for the analysis, it will be deleted.  But as mentioned above,
non-volatile memory has much slower I/O speed than the RAM. Hence in
such situations, the programs will become noticeably slower (sometimes
by factors of 10 times slower, depending on your non-volatile memory
speed).

   Because of the drop in I/O speed (and thus the speed of your running
program), the moment that any to-be-allocated dataset is memory-mapped,
Gnuastro’s programs and libraries will notify you with a descriptive
statement like below (can happen in any phase of their analysis).  It
shows the location of the memory-mapped file, its size, complemented
with a small description of the cause, a pointer to this section of the
book for more information on how to deal with it (if necessary), and
what to do to suppress it.

     astarithmetic: ./gnuastro_mmap/Fu7Dhs: temporary memory-mapped file
     (XXXXXXXXXXX bytes) created for intermediate data that is not stored
     in RAM (see the "Memory management" section of Gnuastro's manual for
     optimizing your project's memory management, and thus speed). To
     disable this warning, please use the option '--quiet-mmap'

Finally, when the intermediate dataset is no longer necessary, the
program will automatically delete it and notify you with a statement
like this:

     astarithmetic: ./gnuastro_mmap/Fu7Dhs: deleted

To disable these messages, you can run the program with ‘--quietmmap’,
or set the ‘quietmmap’ variable in the allocating library function to be
non-zero.

   An important component of these messages is the name of the
memory-mapped file.  Knowing that the file has been deleted is important
for the user if the program crashes for any reason: internally (for
example, a parameter is given wrongly) or externally (for example, you
mistakenly kill the running job).  In the event of a crash, the
memory-mapped files will not be deleted and you have to manually delete
them because they are usually large and they may soon fill your full
storage if not deleted in a long time due to successive crashes.

   This brings us to managing the memory-mapped files in your
non-volatile memory.  In other words: knowing where they are saved, or
intentionally placing them in different places of your file system, or
deleting them when necessary.  As the examples above show, memory-mapped
files are stored in a sub-directory of the running directory called
‘gnuastro_mmap’.  If this directory does not exist, Gnuastro will
automatically create it when memory mapping becomes necessary.
Alternatively, it may happen that the ‘gnuastro_mmap’ sub-directory
exists and is not writable, or it cannot be created.  In such cases, the
memory-mapped file for each dataset will be created in the running
directory with a ‘gnuastro_mmap_’ prefix.

   Therefore one easy way to delete all memory-mapped files in case of a
crash, is to delete everything within the sub-directory (first command
below), or all files stating with this prefix:

     rm -f gnuastro_mmap/*
     rm -f gnuastro_mmap_*

   A much more common issue when dealing with memory-mapped files is
their location.  For example, you may be running a program in a
partition that is hosted by an HDD. But you also have another partition
on an SSD (which has much faster I/O). So you want your memory-mapped
files to be created in the SSD to speed up your processing.  In this
scenario, you want your project source directory to only contain your
plain-text scripts and you want your project’s built products (even the
temporary memory-mapped files) to be built in a different location
because they are large; thus I/O speed becomes important.

   To host the memory-mapped files in another location (with fast I/O),
you can set (‘gnuastro_mmap’) to be a symbolic link to it.  For example,
let’s assume you want your memory-mapped files to be stored in
‘/path/to/dir/for/mmap’.  All you have to do is to run the following
command before your Gnuastro analysis command(s).

     ln -s /path/to/dir/for/mmap gnuastro_mmap

   The programs will delete a memory-mapped file when it is no longer
needed, but they will not delete the ‘gnuastro_mmap’ directory that
hosts them.  So if your project involves many Gnuastro programs
(possibly called in parallel) and you want your memory-mapped files to
be in a different location, you just have to make the symbolic link
above once at the start, and all the programs will use it if necessary.

   Another memory-management scenario that may happen is this: you do
not want a Gnuastro program to allocate internal datasets in the RAM at
all.  For example, the speed of your Gnuastro-related project does not
matter at that moment, and you have higher-priority jobs that are being
run at the same time which need to have RAM available.  In such cases,
you can use the ‘--minmapsize’ option that is available in all Gnuastro
programs (see *note Processing options::).  Any intermediate dataset
that has a size larger than the value of this option will be
memory-mapped, even if there is space available in your RAM. For
example, if you want any dataset larger than 100 megabytes to be
memory-mapped, use ‘--minmapsize=100000000’ (8 zeros!).

   You should not set the value of ‘--minmapsize’ to be too small,
otherwise even small intermediate values (that are usually very
numerous) in the program will be memory-mapped.  However the kernel can
only host a limited number of memory-mapped files at every moment (by
all running programs combined).  For example, in the default(1) Linux
kernel on GNU/Linux operating systems this limit is roughly 64000.  If
the total number of memory-mapped files exceeds this number, all the
programs using them will crash.  Gnuastro’s programs will warn you if
your given value is too small and may cause a problem later.

   Actually, the default behavior for Gnuastro’s programs (to only use
memory-mapped files when there is not enough RAM) is a side-effect of
‘--minmapsize’.  The pre-defined value to this option is an extremely
large value in the lowest-level Gnuastro configuration file (the
installed ‘gnuastro.conf’ described in *note Configuration file
precedence::).  This value is larger than the largest possible available
RAM. You can check by running any Gnuastro program with a ‘-P’ option.
Because no dataset will be larger than this, by default the programs
will first attempt to use the RAM for temporary storage.  But if writing
in the RAM fails (for any reason, mainly due to lack of available
space), then a memory-mapped file will be created.

   ---------- Footnotes ----------

   (1) If you need to host more memory-mapped files at one moment, you
need to build your own customized Linux kernel.


File: gnuastro.info,  Node: Tables,  Next: Tessellation,  Prev: Memory management,  Up: Common program behavior

4.7 Tables
==========

“A table is a collection of related data held in a structured format
within a database.  It consists of columns, and rows.” (from Wikipedia).
Each column in the table contains the values of one property and each
row is a collection of properties (columns) for one target object.  For
example, let’s assume you have just ran MakeCatalog (see *note
MakeCatalog::) on an image to measure some properties for the labeled
regions (which might be detected galaxies for example) in the image.
For each labeled region (detected galaxy), there will be a _row_ which
groups its measured properties as _columns_, one column for each
property.  One such property can be the object’s magnitude, which is the
sum of pixels with that label, or its center can be defined as the
light-weighted average value of those pixels.  Many such properties can
be derived from the raw pixel values and their position, see *note
Invoking astmkcatalog:: for a long list.

   As a summary, for each labeled region (or, galaxy) we have one _row_
and for each measured property we have one _column_.  This high-level
structure is usually the first step for higher-level analysis, for
example, finding the stellar mass or photometric redshift from
magnitudes in multiple colors.  Thus, tables are not just outputs of
programs, in fact it is much more common for tables to be inputs of
programs.  For example, to make a mock galaxy image, you need to feed in
the properties of each galaxy into *note MakeProfiles:: for it do the
inverse of the process above and make a simulated image from a catalog,
see *note Sufi simulates a detection::.  In other cases, you can feed a
table into *note Crop:: and it will crop out regions centered on the
positions within the table, see *note Reddest clumps cutouts and
parallelization::.  So to end this relatively long introduction, tables
play a very important role in astronomy, or generally all branches of
data analysis.

   In *note Recognized table formats:: the currently recognized table
formats in Gnuastro are discussed.  You can use any of these tables as
input or ask for them to be built as output.  The most common type of
table format is a simple plain text file with each row on one line and
columns separated by white space characters, this format is easy to
read/write by eye/hand.  To give it the full functionality of more
specific table types like the FITS tables, Gnuastro has a special
convention which you can use to give each column a name, type, unit, and
comments, while still being readable by other plain text table readers.
This convention is described in *note Gnuastro text table format::.

   When tables are input to a program, the program reading it needs to
know which column(s) it should use for its desired purposes.  Gnuastro’s
programs all follow a similar convention, on the way you can select
columns in a table.  They are thoroughly discussed in *note Selecting
table columns::.

* Menu:

* Recognized table formats::    Table formats that are recognized in Gnuastro.
* Gnuastro text table format::  Gnuastro’s convention plain text tables.
* Selecting table columns::     Identify/select certain columns from a table


File: gnuastro.info,  Node: Recognized table formats,  Next: Gnuastro text table format,  Prev: Tables,  Up: Tables

4.7.1 Recognized table formats
------------------------------

The list of table formats that Gnuastro can currently read from and
write to are described below.  Each has their own advantage and
disadvantages, so a short review of the format is also provided to help
you make the best choice based on how you want to define your input
tables or later use your output tables.

Plain text table
     This is the most basic and simplest way to create, view, or edit
     the table by hand on a text editor.  The other formats described
     below are less eye-friendly and have a more formal structure (for
     easier computer readability).  It is fully described in *note
     Gnuastro text table format::.

FITS ASCII tables
     The FITS ASCII table extension is fully in ASCII encoding and thus
     easily readable on any text editor (assuming it is the only
     extension in the FITS file).  If the FITS file also contains binary
     extensions (for example, an image or binary table extensions), then
     there will be many hard to print characters.  The FITS ASCII format
     does not have new line characters to separate rows.  In the FITS
     ASCII table standard, each row is defined as a fixed number of
     characters (value to the ‘NAXIS1’ keyword), so to visually inspect
     it properly, you would have to adjust your text editor’s width to
     this value.  All columns start at given character positions and
     have a fixed width (number of characters).

     Numbers in a FITS ASCII table are printed into ASCII format, they
     are not in binary (that the CPU uses).  Hence, they can take a
     larger space in memory, loose their precision, and take longer to
     read into memory.  If you are dealing with integer type columns
     (see *note Numeric data types::), another issue with FITS ASCII
     tables is that the type information for the column will be lost
     (there is only one integer type in FITS ASCII tables).  One problem
     with the binary format on the other hand is that it is not portable
     (different CPUs/compilers) have different standards for translating
     the zeros and ones.  But since ASCII characters are defined on a
     byte and are well recognized, they are better for portability on
     those various systems.  Gnuastro’s plain text table format
     described below is much more portable and easier to
     read/write/interpret by humans manually.

     Generally, as the name implies, this format is useful for when your
     table mainly contains ASCII columns (for example, file names, or
     descriptions).  They can be useful when you need to include columns
     with structured ASCII information along with other extensions in
     one FITS file.  In such cases, you can also consider header
     keywords (see *note Fits::).

FITS binary tables
     The FITS binary table is the FITS standard’s solution to the issues
     discussed with keeping numbers in ASCII format as described under
     the FITS ASCII table title above.  Only columns defined as a string
     type (a string of ASCII characters) are readable in a text editor.
     The portability problem with binary formats discussed above is
     mostly solved thanks to the portability of CFITSIO (see *note
     CFITSIO::) and the very long history of the FITS format which has
     been widely used since the 1970s.

     In the case of most numbers, storing them in binary format is more
     memory efficient than ASCII format.  For example, to store
     ‘-25.72034’ in ASCII format, you need 9 bytes/characters.  But if
     you keep this same number (to the approximate precision possible)
     as a 4-byte (32-bit) floating point number, you can keep/transmit
     it with less than half the amount of memory.  When catalogs contain
     thousands/millions of rows in tens/hundreds of columns, this can
     lead to significant improvements in memory/band-width usage.
     Moreover, since the CPU does its operations in the binary formats,
     reading the table in and writing it out is also much faster than an
     ASCII table.

     When you are dealing with integer numbers, the compression ratio
     can be even better, for example, if you know all of the values in a
     column are positive and less than ‘255’, you can use the ‘unsigned
     char’ type which only takes one byte!  If they are between ‘-128’
     and ‘127’, then you can use the (signed) ‘char’ type.  So if you
     are thoughtful about the limits of your integer columns, you can
     greatly reduce the size of your file and also the speed at which it
     is read/written.  This can be very useful when sharing your results
     with collaborators or publishing them.  To decrease the file size
     even more you can name your output as ending in ‘.fits.gz’ so it is
     also compressed after creation.  Just note that
     compression/decompressing is CPU intensive and can slow down the
     writing/reading of the file.

     Fortunately the FITS Binary table format also accepts ASCII strings
     as column types (along with the various numerical types).  So your
     dataset can also contain non-numerical columns.

* Menu:

* Gnuastro text table format::  Reading plain text tables


File: gnuastro.info,  Node: Gnuastro text table format,  Next: Selecting table columns,  Prev: Recognized table formats,  Up: Tables

4.7.2 Gnuastro text table format
--------------------------------

Plain text files are the most generic, portable, and easiest way to
(manually) create, (visually) inspect, or (manually) edit a table.  In
this format, the ending of a row is defined by the new-line character (a
line on a text editor).  So when you view it on a text editor, every row
will occupy one line.  The delimiters (or characters separating the
columns) are white space characters (space, horizontal tab, vertical
tab) and a comma (<,>).  The only further requirement is that all
rows/lines must have the same number of columns.

   The columns do not have to be exactly under each other and the rows
can be arbitrarily long with different lengths.  For example, the
following contents in a file would be interpreted as a table with 4
columns and 2 rows, with each element interpreted as a 64-bit floating
point type (see *note Numeric data types::).

     1     2.234948   128   39.8923e8
     2 , 4.454        792     72.98348e7

   However, the example above has no other information about the columns
(it is just raw data, with no meta-data).  To use this table, you have
to remember what the numbers in each column represent.  Also, when you
want to select columns, you have to count their position within the
table.  This can become frustrating and prone to bad errors (getting the
columns wrong in your scientific project!)  especially as the number of
columns increase.  It is also bad for sending to a colleague, because
they will find it hard to remember/use the columns properly.

   To solve these problems in Gnuastro’s programs/libraries you are not
limited to using the column’s number, see *note Selecting table
columns::.  If the columns have names, units, or comments you can also
select your columns based on searches/matches in these fields, for
example, see *note Table::.  Also, in this manner, you cannot guide the
program reading the table on how to read the numbers.  As an example,
the first and third columns above can be read as integer types: the
first column might be an ID and the third can be the number of pixels an
object occupies in an image.  So there is no need to read these to
columns as a 64-bit floating point type (which takes more memory, and is
slower).

   In the bare-minimum example above, you also cannot use strings of
characters, for example, the names of filters, or some other identifier
that includes non-numerical characters.  In the absence of any
information, only numbers can be read robustly.  Assuming we read
columns with non-numerical characters as string, there would still be
the problem that the strings might contain space (or any delimiter)
character for some rows.  So, each ‘word’ in the string will be
interpreted as a column and the program will abort with an error that
the rows do not have the same number of columns.

   To correct for these limitations, Gnuastro defines the following
convention for storing the table meta-data along with the raw data in
one plain text file.  The format is primarily designed for ease of
reading/writing by eye/fingers, but is also structured enough to be read
by a program.

   When the first non-white character in a line is <#>, or there are no
non-white characters in it, then the line will not be considered as a
row of data in the table (this is a pretty standard convention in many
programs, and higher level languages).  In the first case (when the
first character of the line is <#>), the line is interpreted as a
_comment_.

   If the comment line starts with ‘‘# Column N:’’, then it is assumed
to contain information about column ‘N’ (a number, counting from 1).
Comment lines that do not start with this pattern are ignored and you
can use them to include any further information you want to store with
the table in the text file.  The most generic column information comment
line has the following format:

     # Column N: NAME [UNIT, TYPE(NUM), BLANK] COMMENT

Any sequence of characters between ‘<:>’ and ‘<[>’ will be interpreted
as the column name (so it can contain anything except the ‘<[>’
character).  Anything between the ‘<]>’ and the end of the line is
defined as a comment.  Within the brackets, anything before the first
‘<,>’ is the units (physical units, for example, km/s, or erg/s),
anything before the second ‘<,>’ is the short type identifier (see
below, and *note Numeric data types::).

   If the type identifier is not recognized, the default 64-bit floating
point type will be used.  The type identifier can optionally be followed
by an integer within parenthesis.  If the parenthesis is present and the
integer is larger than 1, the column is assumed to be a “vector column”
(which can have multiple values, for more see *note Vector columns::).

   Finally (still within the brackets), any non-white characters after
the second ‘<,>’ are interpreted as the blank value for that column (see
*note Blank pixels::).  The blank value can either be in the same type
as the column (for example, ‘-99’ for a signed integer column), or any
string (for example, ‘NaN’ in that same column).  In both cases, the
values will be stored in memory as Gnuastro’s fixed blank values for
each type.  For floating point types, Gnuastro’s internal blank value is
IEEE NaN (Not-a-Number).  For signed integers, it is the smallest
possible value and for unsigned integers its the largest possible value.

   When a formatting problem occurs, or when the column was already
given meta-data in a previous comment, or when the column number is
larger than the actual number of columns in the table (the non-commented
or empty lines), then the comment information line will be ignored.

   When a comment information line can be used, the leading and trailing
white space characters will be stripped from all of the elements.  For
example, in this line:

     # Column 5:  column name   [km/s,    f32,-99] Redshift as speed

   The ‘NAME’ field will be ‘‘column name’’ and the ‘TYPE’ field will be
‘‘f32’’.  Note how all the white space characters before and after
strings are not used, but those in the middle remained.  Also, white
space characters are not mandatory.  Hence, in the example above, the
‘BLANK’ field will be given the value of ‘‘-99’’.

   Except for the column number (‘N’), the rest of the fields are
optional.  Also, the column information comments do not have to be in
order.  In other words, the information for column $N+m$ ($m>0$) can be
given in a line before column $N$.  Furthermore, you do not have to
specify information for all columns.  Those columns that do not have
this information will be interpreted with the default settings (like the
case above: values are double precision floating point, and the column
has no name, unit, or comment).  So these lines are all acceptable for
any table (the first one, with nothing but the column number is
redundant):

     # Column 5:
     # Column 1: ID [,i8] The Clump ID.
     # Column 3: mag_f160w [AB mag, f32] Magnitude from the F160W filter

The data type of the column should be specified with one of the
following values:

   • For a numeric column, you can use any of the numeric types (and
     their recognized identifiers) described in *note Numeric data
     types::.
   • ‘‘strN’’: for strings.  The ‘N’ value identifies the length of the
     string (how many characters it has).  The start of the string on
     each row is the first non-delimiter character of the column that
     has the string type.  The next ‘N’ characters will be interpreted
     as a string and all leading and trailing white space will be
     removed.

     If the next column’s characters, are closer than ‘N’ characters to
     the start of the string column in that line/row, they will be
     considered part of the string column.  If there is a new-line
     character before the ending of the space given to the string column
     (in other words, the string column is the last column), then
     reading of the string will stop, even if the ‘N’ characters are not
     complete yet.  See ‘tests/table/table.txt’ for one example.
     Therefore, the only time you have to pay attention to the
     positioning and spaces given to the string column is when it is not
     the last column in the table.

     The only limitation in this format is that trailing and leading
     white space characters will be removed from the columns that are
     read.  In most cases, this is the desired behavior, but if trailing
     and leading white-spaces are critically important to your analysis,
     define your own starting and ending characters and remove them
     after the table has been read.  For example, in the sample table
     below, the two ‘<|>’ characters (which are arbitrary) will remain
     in the value of the second column and you can remove them manually
     later.  If only one of the leading or trailing white spaces is
     important for your work, you can only use one of the ‘<|>’s.

          # Column 1: ID [label, u8]
          # Column 2: Notes [no unit, str50]
          1    leading and trailing white space is ignored here    2.3442e10
          2   |         but they will be preserved here        |   8.2964e11

   Note that the FITS binary table standard does not define the
‘unsigned int’ and ‘unsigned long’ types, so if you want to convert your
tables to FITS binary tables, use other types.  Also, note that in the
FITS ASCII table, there is only one integer type (‘long’).  So if you
convert a Gnuastro plain text table to a FITS ASCII table with the *note
Table:: program, the type information for integers will be lost.
Conversely if integer types are important for you, you have to manually
set them when reading a FITS ASCII table (for example, with the Table
program when reading/converting into a file, or with the
‘gnuastro/table.h’ library functions when reading into memory).


File: gnuastro.info,  Node: Selecting table columns,  Prev: Gnuastro text table format,  Up: Tables

4.7.3 Selecting table columns
-----------------------------

At the lowest level, the only defining aspect of a column in a table is
its number, or position.  But selecting columns purely by number is not
very convenient and, especially when the tables are large it can be very
frustrating and prone to errors.  Hence, table file formats (for
example, see *note Recognized table formats::) have ways to store
additional information about the columns (meta-data).  Some of the most
common pieces of information about each column are its _name_, the
_units_ of data in it, and a _comment_ for longer/informal description
of the column’s data.

   To facilitate research with Gnuastro, you can select columns by
matching, or searching in these three fields, besides the low-level
column number.  To view the full list of information on the columns in
the table, you can use the Table program (see *note Table::) with the
command below (replace ‘table-file’ with the filename of your table, if
its FITS, you might also need to specify the HDU/extension which
contains the table):

     $ asttable --information table-file

   Gnuastro’s programs need the columns for different purposes, for
example, in Crop, you specify the columns containing the central
coordinates of the crop centers with the ‘--coordcol’ option (see *note
Crop options::).  On the other hand, in MakeProfiles, to specify the
column containing the profile position angles, you must use the ‘--pcol’
option (see *note MakeProfiles catalog::).  Thus, there can be no
unified common option name to select columns for all programs (different
columns have different purposes).  However, when the program expects a
column for a specific context, the option names end in the ‘col’ suffix
like the examples above.  These options accept values in integer (column
number), or string (metadata match/search) format.

   If the value can be parsed as a positive integer, it will be seen as
the low-level column number.  Note that column counting starts from 1,
so if you ask for column 0, the respective program will abort with an
error.  When the value cannot be interpreted as an a integer number, it
will be seen as a string of characters which will be used to
match/search in the table’s meta-data.  The meta-data field which the
value will be compared with can be selected through the ‘--searchin’
option, see *note Input output options::.  ‘--searchin’ can take three
values: ‘name’, ‘unit’, ‘comment’.  The matching will be done following
this convention:

   • If the value is enclosed in two slashes (for example, ‘-x/RA_/’, or
     ‘--coordcol=/RA_/’, see *note Crop options::), then it is assumed
     to be a regular expression with the same convention as GNU AWK. GNU
     AWK has a very well written chapter
     (https://www.gnu.org/software/gawk/manual/html_node/Regexp.html)
     describing regular expressions, so we will not continue discussing
     them here.  Regular expressions are a very powerful tool in
     matching text and useful in many contexts.  We thus strongly
     encourage reviewing this chapter for greatly improving the quality
     of your work in many cases, not just for searching column meta-data
     in Gnuastro.

   • When the string is not enclosed between ‘</>’s, any column that
     exactly matches the given value in the given field will be
     selected.

   Note that in both cases, you can ignore the case of alphabetic
characters with the ‘--ignorecase’ option, see *note Input output
options::.  Also, in both cases, multiple columns may be selected with
one call to this function.  In this case, the order of the selected
columns (with one call) will be the same order as they appear in the
table.


File: gnuastro.info,  Node: Tessellation,  Next: Automatic output,  Prev: Tables,  Up: Common program behavior

4.8 Tessellation
================

It is sometimes necessary to classify the elements in a dataset (for
example, pixels in an image) into a grid of individual, non-overlapping
tiles.  For example, when background sky gradients are present in an
image, you can define a tile grid over the image.  When the tile sizes
are set properly, the background’s variation over each tile will be
negligible, allowing you to measure (and subtract) it.  In other cases
(for example, spatial domain convolution in Gnuastro, see *note
Convolve::), it might simply be for speed of processing: each tile can
be processed independently on a separate CPU thread.  In the arts and
mathematics, this process is formally known as tessellation
(https://en.wikipedia.org/wiki/Tessellation).

   The size of the regular tiles (in units of data-elements, or pixels
in an image) can be defined with the ‘--tilesize’ option.  It takes
multiple numbers (separated by a comma) which will be the length along
the respective dimension (in FORTRAN/FITS dimension order).  Divisions
are also acceptable, but must result in an integer.  For example,
‘--tilesize=30,40’ can be used for an image (a 2D dataset).  The regular
tile size along the first FITS axis (horizontal when viewed in SAO DS9)
will be 30 pixels and along the second it will be 40 pixels.  Ideally,
‘--tilesize’ should be selected such that all tiles in the image have
exactly the same size.  In other words, that the dataset length in each
dimension is divisible by the tile size in that dimension.

   However, this is not always possible: the dataset can be any size and
every pixel in it is valuable.  In such cases, Gnuastro will look at the
significance of the remainder length, if it is not significant (for
example, one or two pixels), then it will just increase the size of the
first tile in the respective dimension and allow the rest of the tiles
to have the required size.  When the remainder is significant (for
example, one pixel less than the size along that dimension), the
remainder will be added to one regular tile’s size and the large tile
will be cut in half and put in the two ends of the grid/tessellation.
In this way, all the tiles in the central regions of the dataset will
have the regular tile sizes and the tiles on the edge will be slightly
larger/smaller depending on the remainder significance.  The fraction
which defines the remainder significance along all dimensions can be set
through ‘--remainderfrac’.

   The best tile size is directly related to the spatial properties of
the property you want to study (for example, gradient on the image).  In
practice we assume that the gradient is not present over each tile.  So
if there is a strong gradient (for example, in long wavelength ground
based images) or the image is of a crowded area where there is not too
much blank area, you have to choose a smaller tile size.  A larger mesh
will give more pixels and so the scatter in the results will be less
(better statistics).

   For raw image processing, a single tessellation/grid is not
sufficient.  Raw images are the unprocessed outputs of the camera
detectors.  Modern detectors usually have multiple readout channels each
with its own amplifier.  For example, the Hubble Space Telescope
Advanced Camera for Surveys (ACS) has four amplifiers over its full
detector area dividing the square field of view to four smaller squares.
Ground based image detectors are not exempt, for example, each CCD of
Subaru Telescope’s Hyper Suprime-Cam camera (which has 104 CCDs) has
four amplifiers, but they have the same height of the CCD and divide the
width by four parts.

   The bias current on each amplifier is different, and initial bias
subtraction is not perfect.  So even after subtracting the measured bias
current, you can usually still identify the boundaries of different
amplifiers by eye.  See Figure 11(a) in Akhlaghi and Ichikawa (2015) for
an example.  This results in the final reduced data to have non-uniform
amplifier-shaped regions with higher or lower background flux values.
Such systematic biases will then propagate to all subsequent
measurements we do on the data (for example, photometry and subsequent
stellar mass and star formation rate measurements in the case of
galaxies).

   Therefore an accurate analysis requires a two layer tessellation: the
top layer contains larger tiles, each covering one amplifier channel.
For clarity we will call these larger tiles “channels”.  The number of
channels along each dimension is defined through the ‘--numchannels’.
Each channel is then covered by its own individual smaller tessellation
(with tile sizes determined by the ‘--tilesize’ option).  This will
allow independent analysis of two adjacent pixels from different
channels if necessary.  If the image is processed or the detector only
has one amplifier, you can set the number of channels in both dimension
to 1.

   The final tessellation can be inspected on the image with the
‘--checktiles’ option that is available to all programs which use
tessellation for localized operations.  When this option is called, a
FITS file with a ‘_tiled.fits’ suffix will be created along with the
outputs, see *note Automatic output::.  Each pixel in this image has the
number of the tile that covers it.  If the number of channels in any
dimension are larger than unity, you will notice that the tile IDs are
defined such that the first channels is covered first, then the second
and so on.  For the full list of processing-related common options
(including tessellation options), please see *note Processing options::.


File: gnuastro.info,  Node: Automatic output,  Next: Output FITS files,  Prev: Tessellation,  Up: Common program behavior

4.9 Automatic output
====================

All the programs in Gnuastro are designed such that specifying an output
file or directory (based on the program context) is optional.  When no
output name is explicitly given (with ‘--output’, see *note Input output
options::), the programs will automatically set an output name based on
the input name(s) and what the program does.  For example, when you are
using ConvertType to save FITS image named ‘dataset.fits’ to a JPEG
image and do not specify a name for it, the JPEG output file will be
name ‘dataset.jpg’.  When the input is from the standard input (for
example, a pipe, see *note Standard input::), and ‘--output’ is not
given, the output name will be the program’s name (for example,
‘converttype.jpg’).

   Another very important part of the automatic output generation is
that all the directory information of the input file name is stripped
off of it.  This feature can be disabled with the ‘--keepinputdir’
option, see *note Input output options::.  It is the default because
astronomical data are usually very large and organized specially with
special file names.  In some cases, the user might not have write
permissions in those directories(1).

   Let’s assume that we are working on a report and want to process the
FITS images from two projects (ABC and DEF), which are stored in the
sub-directories named ‘ABCproject/’ and ‘DEFproject/’ of our top data
directory (‘/mnt/data’).  The following shell commands show how one
image from the former is first converted to a JPEG image through
ConvertType and then the objects from an image in the latter project are
detected using NoiseChisel.  The text after the ‘#’ sign are comments
(not typed!).

     $ pwd                                               # Current location
     /home/usrname/research/report
     $ ls                                         # List directory contents
     ABC01.jpg
     $ ls /mnt/data/ABCproject                                  # Archive 1
     ABC01.fits ABC02.fits ABC03.fits
     $ ls /mnt/data/DEFproject                                  # Archive 2
     DEF01.fits DEF02.fits DEF03.fits
     $ astconvertt /mnt/data/ABCproject/ABC02.fits --output=jpg    # Prog 1
     $ ls
     ABC01.jpg ABC02.jpg
     $ astnoisechisel /mnt/data/DEFproject/DEF01.fits              # Prog 2
     $ ls
     ABC01.jpg ABC02.jpg DEF01_detected.fits

   ---------- Footnotes ----------

   (1) In fact, even if the data is stored on your own computer, it is
advised to only grant write permissions to the super user or root.  This
way, you will not accidentally delete or modify your valuable data!


File: gnuastro.info,  Node: Output FITS files,  Next: Numeric locale,  Prev: Automatic output,  Up: Common program behavior

4.10 Output FITS files
======================

The output of many of Gnuastro’s programs are (or can be) FITS files.
The FITS format has many useful features for storing scientific datasets
(cubes, images and tables) along with a robust features for
archivability.  For more on this standard, please see *note Fits::.

   As a community convention described in *note Fits::, the first
extension of all FITS files produced by Gnuastro’s programs only
contains the meta-data that is intended for the file’s extension(s).
For a Gnuastro program, this generic meta-data (that is stored as FITS
keyword records) is its configuration when it produced this dataset:
file name(s) of input(s) and option names, values and comments.  Note
that when the configuration is too trivial (only input filename, for
example, the program *note Table::) no meta-data is written in this
extension.

   FITS keywords have the following limitations in regards to generic
option names and values which are described below:

   • If a keyword (option name) is longer than 8 characters, the first
     word in the record (80 character line) is ‘HIERARCH’ which is
     followed by the keyword name.

   • Values can be at most 75 characters, but for strings, this changes
     to 73 (because of the two extra <'> characters that are necessary).
     However, if the value is a file name, containing slash (</>)
     characters to separate directories, Gnuastro will break the value
     into multiple keywords.

   • Keyword names ignore case, therefore they are all in capital
     letters.  Therefore, if you want to use Grep to inspect these
     keywords, use the ‘-i’ option, like the example below.

          $ astfits image_detected.fits -h0 | grep -i snquant

   The keywords above are classified (separated by an empty line and
title) as a group titled “ProgramName configuration”.  This meta-data
extension, as well as all the other extensions (which contain data),
also contain have final group of keywords to keep the basic date and
version information of Gnuastro, its dependencies and the pipeline that
is using Gnuastro (if it is under version control).

‘DATE’
     The creation time of the FITS file.  This date is written directly
     by CFITSIO and is in UT format.

‘COMMIT’
     Git’s commit description from the running directory of Gnuastro’s
     programs.  If the running directory is not version controlled or
     ‘libgit2’ is not installed (see *note Optional dependencies::) then
     this keyword will not be present.  The printed value is equivalent
     to the output of the following command:

          git describe --dirty --always

     If the running directory contains non-committed work, then the
     stored value will have a ‘‘-dirty’’ suffix.  This can be very
     helpful to let you know that the data is not ready to be shared
     with collaborators or submitted to a journal.  You should only
     share results that are produced after all your work is committed
     (safely stored in the version controlled history and thus
     reproducible).

     At first sight, version control appears to be mainly a tool for
     software developers.  However progress in a scientific research is
     almost identical to progress in software development: first you
     have a rough idea that starts with handful of easy steps.  But as
     the first results appear to be promising, you will have to extend,
     or generalize, it to make it more robust and work in all the
     situations your research covers, not just your first test samples.
     Slowly you will find wrong assumptions or bad implementations that
     need to be fixed (‘bugs’ in software development parlance).
     Finally, when you submit the research to your collaborators or a
     journal, many comments and suggestions will come in, and you have
     to address them.

     Software developers have created version control systems precisely
     for this kind of activity.  Each significant moment in the
     project’s history is called a “commit”, see *note Version
     controlled source::.  A snapshot of the project in each “commit” is
     safely stored away, so you can revert back to it at a later time,
     or check changes/progress.  This way, you can be sure that your
     work is reproducible and track the progress and history.  With
     version control, experimentation in the project’s analysis is
     greatly facilitated, since you can easily revert back if a
     brainstorm test procedure fails.

     One important feature of version control is that the research
     result (FITS image, table, report or paper) can be stamped with the
     unique commit information that produced it.  This information will
     enable you to exactly reproduce that same result later, even if you
     have made changes/progress.  For one example of a research paper’s
     reproduction pipeline, please see the reproduction pipeline
     (https://gitlab.com/makhlaghi/NoiseChisel-paper) of the paper
     (https://arxiv.org/abs/1505.01664) describing *note NoiseChisel::.

‘CFITSIO’
     The version of CFITSIO used (see *note CFITSIO::).

‘WCSLIB’
     The version of WCSLIB used (see *note WCSLIB::).  Note that older
     versions of WCSLIB do not report the version internally.  So this
     is only available if you are using more recent WCSLIB versions.

‘GSL’
     The version of GNU Scientific Library that was used, see *note GNU
     Scientific Library::.

‘GNUASTRO’
     The version of Gnuastro used (see *note Version numbering::).

   Here is one example of the last few lines of an example output.

                   / Versions and date
     DATE    = '...'                / file creation date
     COMMIT  = 'v0-8-g547f6eb'      / Commit description in running dir.
     CFITSIO = '3.45    '           / CFITSIO version.
     WCSLIB  = '5.19    '           / WCSLIB version.
     GSL     = '2.5     '           / GNU Scientific Library version.
     GNUASTRO= '0.7     '           / GNU Astronomy Utilities version.
     END


File: gnuastro.info,  Node: Numeric locale,  Prev: Output FITS files,  Up: Common program behavior

4.11 Numeric locale
===================

If your system locale
(https://en.wikipedia.org/wiki/Locale_(computer_software)) is not
English, it may happen that the ‘.’ is not used as the decimal separator
of basic command-line tools for input or output.  For example, in
Spanish and some other languages the decimal separator (symbol used to
separate the integer and fractional part of a number), is a comma.
Therefore in such systems, some programs may print $0.5$ as as ‘‘0,5’’
(instead of ‘‘0.5’’).  This mainly happens in some core operating system
tools like ‘awk’ or ‘seq’ depend on the locale.  This can cause problems
for other programs (like those in Gnuastro that expect a ‘<.>’ as the
decimal separator).

   To see the effect, please try the commands below.  The first one will
print $0.5$ in your default locale’s format.  The second set will use
the Spanish locale for printing numbers (which will put a comma between
the 0 and the 5).  The third will use the English (US) locale for
printing numbers (which will put a point between the 0 and the 5).

     $ seq 0.5 1

     $ export LC_NUMERIC=es_ES.utf8
     $ seq 0.5 1

     $ export LC_NUMERIC=en_US.utf8
     $ seq 0.5 1

With the simple command below, you can check your current locale
environment variables for specifying the formats of various things like
date, time, monetary, telephone, numbers, etc.  You can change any of
these, by simply giving different values to the respective variable like
above.  For a more complete explanation on each variable, see
<https://www.baeldung.com/linux/locale-environment-variables>.

     $ locale

   To avoid these kinds of locale-specific problems (for example,
another program not being able to read ‘‘0,5’’ as half of unity), you
can change the locale by giving the value of ‘C’ to the ‘LC_NUMERIC’
environment variable (or the lower-level/generic ‘LC_ALL’).  You will
notice that ‘C’ is not a human-language and country identifier like
‘en_US’, it is the programming locale, which is well recognized by
programmers in all countries and is available on all Unix-like operating
systems (others may not be pre-defined and may need installation).  You
can set the ‘LC_NUMERIC’ only for a single command (the first one below:
simply defining the variable in the same line), or all commands within
the running session (the second command below, or “exporting” it to all
subsequent commands):

     ## Change the numeric locale, only for this 'seq' command.
     $ LC_NUMERIC=C seq 0.5 1

     ## Change the locale to the standard, for all commands after it.
     $ export LC_NUMERIC=C

   If you want to change it generally for all future sessions, you can
put the second command in your shell’s startup file.  For more on
startup files, please see *note Installation directory::.


File: gnuastro.info,  Node: Data containers,  Next: Data manipulation,  Prev: Common program behavior,  Up: Top

5 Data containers
*****************

The most low-level and basic property of a dataset is how it is stored.
To process, archive and transmit the data, you need a container to store
it first.  From the start of the computer age, different formats have
been defined to store data, optimized for particular applications.  One
format/container can never be useful for all applications: the storage
defines the application and vice-versa.  In astronomy, the Flexible
Image Transport System (FITS) standard has become the most common format
of data storage and transmission.  It has many useful features, for
example, multiple sub-containers (also known as extensions or header
data units, HDUs) within one file, or support for tables as well as
images.  Each HDU can store an independent dataset and its corresponding
meta-data.  Therefore, Gnuastro has one program (see *note Fits::)
specifically designed to manipulate FITS HDUs and the meta-data (header
keywords) in each HDU.

   Your astronomical research does not just involve data analysis (where
the FITS format is very useful).  For example, you want to demonstrate
your raw and processed FITS images or spectra as figures within slides,
reports, or papers.  The FITS format is not defined for such
applications.  Thus, Gnuastro also comes with the ConvertType program
(see *note ConvertType::) which can be used to convert a FITS image to
and from (where possible) other formats like plain text and JPEG (which
allow two way conversion), along with EPS and PDF (which can only be
created from FITS, not the other way round).

   Finally, the FITS format is not just for images, it can also store
tables.  Binary tables in particular can be very efficient in storing
catalogs that have more than a few tens of columns and rows.  However,
unlike images (where all elements/pixels have one data type), tables
contain multiple columns and each column can have different properties:
independent data types (see *note Numeric data types::) and meta-data.
In practice, each column can be viewed as a separate container that is
grouped with others in the table.  The only shared property of the
columns in a table is thus the number of elements they contain.  To
allow easy inspection/manipulation of table columns, Gnuastro has the
Table program (see *note Table::).  It can be used to select certain
table columns in a FITS table and see them as a human readable output on
the command-line, or to save them into another plain text or FITS table.

* Menu:

* Fits::                        View and manipulate extensions and keywords.
* ConvertType::                 Convert data to various formats.
* Table::                       Read and Write FITS tables to plain text.
* Query::                       Import data from external databases.


File: gnuastro.info,  Node: Fits,  Next: ConvertType,  Prev: Data containers,  Up: Data containers

5.1 Fits
========

The “Flexible Image Transport System”, or FITS, is by far the most
common data container format in astronomy and in constant use since the
1970s.  Archiving (future usage, simplicity) has been one of the primary
design principles of this format.  In the last few decades it has proved
so useful and robust that the Vatican Library has also chosen FITS for
its “long-term digital preservation” project(1).

   Although the full name of the standard invokes the idea that it is
only for images, it also contains complete and robust features for
tables.  It started off in the 1970s and was formally published as a
standard in 1981, it was adopted by the International Astronomical Union
(IAU) in 1982 and an IAU working group to maintain its future was
defined in 1988.  The FITS 2.0 and 3.0 standards were approved in 2000
and 2008 respectively, and the 4.0 draft has also been released
recently, please see the FITS standard document web page
(https://fits.gsfc.nasa.gov/fits_standard.html) for the full text of all
versions.  Also see the FITS 3.0 standard paper
(https://doi.org/10.1051/0004-6361/201015362) for a nice introduction
and history along with the full standard.

   Many common image formats, for example, a JPEG, only have one
image/dataset per file, however one great advantage of the FITS standard
is that it allows you to keep multiple datasets (images or tables along
with their separate meta-data) in one file.  In the FITS standard, each
data + metadata is known as an extension, or more formally a header data
unit or HDU. The HDUs in a file can be completely independent: you can
have multiple images of different dimensions/sizes or tables as separate
extensions in one file.  However, while the standard does not impose any
constraints on the relation between the datasets, it is strongly
encouraged to group data that are contextually related with each other
in one file.  For example, an image and the table/catalog of objects and
their measured properties in that image.  Other examples can be images
of one patch of sky in different colors (filters), or one raw telescope
image along with its calibration data (tables or images).

   As discussed above, the extensions in a FITS file can be completely
independent.  To keep some information (meta-data) about the group of
extensions in the FITS file, the community has adopted the following
convention: put no data in the first extension, so it is just meta-data.
This extension can thus be used to store Meta-data regarding the whole
file (grouping of extensions).  Subsequent extensions may contain data
along with their own separate meta-data.  All of Gnuastro’s programs
also follow this convention: the main output dataset(s) are placed in
the second (or later) extension(s).  The first extension contains no
data the program’s configuration (input file name, along with all its
option values) are stored as its meta-data, see *note Output FITS
files::.

   The meta-data contain information about the data, for example, which
region of the sky an image corresponds to, the units of the data, what
telescope, camera, and filter the data were taken with, it observation
date, or the software that produced it and its configuration.  Without
the meta-data, the raw dataset is practically just a collection of
numbers and really hard to understand, or connect with the real world
(other datasets).  It is thus strongly encouraged to supplement your
data (at any level of processing) with as much meta-data about your
processing/science as possible.

   The meta-data of a FITS file is in ASCII format, which can be easily
viewed or edited with a text editor or on the command-line.  Each
meta-data element (known as a keyword generally) is composed of a name,
value, units and comments (the last two are optional).  For example,
below you can see three FITS meta-data keywords for specifying the world
coordinate system (WCS, or its location in the sky) of a dataset:

     LATPOLE =           -27.805089 / [deg] Native latitude of celestial pole
     RADESYS = 'FK5'                / Equatorial coordinate system
     EQUINOX =               2000.0 / [yr] Equinox of equatorial coordinates

   However, there are some limitations which discourage viewing/editing
the keywords with text editors.  For example, there is a fixed length of
80 characters for each keyword (its name, value, units and comments) and
there are no new-line characters, so on a text editor all the keywords
are seen in one line.  Also, the meta-data keywords are immediately
followed by the data which are commonly in binary format and will show
up as strange looking characters on a text editor, and significantly
slowing down the processor.

   Gnuastro’s Fits program was designed to allow easy manipulation of
FITS extensions and meta-data keywords on the command-line while
conforming fully with the FITS standard.  For example, you can copy or
cut (copy and remove) HDUs/extensions from one FITS file to another, or
completely delete them.  It also has features to delete, add, or edit
meta-data keywords within one HDU.

* Menu:

* Invoking astfits::            Arguments and options to Header.

   ---------- Footnotes ----------

   (1) <https://www.vaticanlibrary.va/home.php?pag=progettodigit>

