\input texinfo    @c -*-texinfo-*-

@c %**start of header

@set VERSION %%%%PACKAGE_VERSION%%%%

@setfilename tablicious.info
@settitle Octave Tablicious v@value{VERSION}
@documentencoding UTF-8

@c %**end of header

@copying
This manual is for Tablicious, version @value{VERSION}.

Copyright @copyright{} 2019 Andrew Janke

@quotation
Permission is granted to make and distribute verbatim copies of
this manual provided the copyright notice and this permission notice
are preserved on all copies.

Permission is granted to copy and distribute modified versions of this
manual under the conditions for verbatim copying, provided that the entire
resulting derived work is distributed under the terms of a permission
notice identical to this one.

Permission is granted to copy and distribute translations of this manual
into another language, under the same conditions as for modified versions.
@end quotation
@end copying

@documentdescription
User manual for Tablicious for Octave.
@end documentdescription

@titlepage
@title  Tablicious for Octave
@subtitle version @value{VERSION}, %%%%PACKAGE_DATE_YEARMONTH%%%%
@author Andrew Janke
@page
@vskip 0pt plus 1filll
@insertcopying
@end titlepage

@shortcontents
@contents

@ifnottex
@node Top, Introduction
@top Tablicious for GNU Octave

This manual is for Tablicious, version @value{VERSION}.
@end ifnottex

@menu
* Introduction::
* Getting Started::
* Table Representation::
* Date and Time Representation::
* Validation Functions::
* Example Data Sets::
* Data Sets from R::
* Missing Functionality::
* API Reference::
* Copying::
@end menu


@node Introduction
@chapter Introduction

@quotation
Time is an illusion. Lunchtime doubly so.

@author Douglas Adams
@end quotation


This is the manual for the Tablicious package version @value{VERSION} for GNU Octave.

Tablicious provides Matlab-compatible tabular data and date/time support for GNU Octave.
This includes a @code{table} class with support for filtering and join operations;
@code{datetime}, @code{duration}, and related classes;
Missing Data support; @code{string} and @code{categorical} data types; 
and other miscellaneous things.

This document is a work in progress. You are invited to help improve it and
submit patches.

Tablicious’s classes are designed to be convenient to use while still being efficient.
The data representations used by Tablicious are designed to be efficient and suitable
for working with large-ish data sets. A “large-ish” data set is one that can have
millions of elements or rows, but still fits in main computer memory. Tablicious’s main
relational and arithmetic operations are all implemented using vectorized
operations on primitive Octave data types.

Tablicious was written by Andrew Janke <@email{floss@@apjanke.net}>. Support can be
found on the @url{https://github.com/apjanke/octave-tablicious, Tablicious project
GitHub page}.


@node Getting Started
@chapter Getting Started

The easiest way to obtain Tablicious is by using Octave's @code{pkg} package manager. 
To install the development prerelease of Tablicious, run this in Octave:

@example
pkg install https://github.com/apjanke/octave-tablicious/releases/download/v%%%%PACKAGE_VERSION%%%%/tablicious-%%%%PACKAGE_VERSION%%%%.tar.gz
@end example

(Check the releases page at @url{https://github.com/apjanke/octave-tablicious/releases} to
find out what the actual latest release number is.)

For development, you can obtain the source code for Tablicious from the project repo on GitHub at 
@url{https://github.com/apjanke/octave-tablicious}. Make a local clone of the repo.
Then add the @file{inst} directory in the repo to your Octave path.



@node Table Representation
@chapter Table Representation

Tablicious provides the @code{table} class for representing tabular data.

A @code{table} is an array object that represents a tabular data structure. It holds
multiple named “variables”, each of which is a column vector, or a 2-D matrix whose
rows are read as records.

A @code{table} is composed of multiple “variables”, each with a name, which all have
the same number of rows. (A @code{table} variable is like a “column” in SQL tables
or in R or Python/pandas dataframes. Whenever you read “variable” here, think
“column”.) Taken together, the @var{i}-th element or row of each variable compose
a single record or observation.

Tables are good ways of arranging data if you have data that would otherwise be stored
in a few separate variables which all need to be kept in the same shape and order,
especially if you might want to do element-wise comparisons involving two or more of
those variables. That’s basically all a @code{table} is: it holds a collection of
variables, and makes sure they are all kept aligned and ordered in the same way.

Tables are a lot like SQL tables or result sets, and are based on the same relational
algebra theory that SQL is. Many common, even powerful, SQL operations can be done
in Octave using @code{table} arrays. It’s like having your own in-memory SQL engine.

@node Table Construction
@section Table Construction

There are two main ways to construct a @code{table} array: build one up by combining
multiple variables together, or convert an existing tabular-organized array into a 
@code{table}.

To build an array from multiple variables, use the @code{table(@dots{})} constructor, passing
in all of your variables as separate inputs. It takes any number of inputs. Each input
becomes a table variable in the new @code{table} object. If you pass your constructor
inputs directly from variables, it automatically picks up their names and uses them
as the table variable names. Otherwise, if you’re using more complex expressions, you’ll
need to supply the @code{'VariableNames'} option.

To convert a tabular-organized array of another type into a @code{table}, use the 
conversion functions like @code{array2table}, @code{struct2table} and @code{cell2table}.
@code{array2table} and @code{cell2table} take each column of the input array and turn
it into a separate table variable in the resulting @code{table}. @code{struct2table} takes
the fields of a struct and puts them into table variables.

@node Tables vs SQL
@section Tables vs SQL

Here’s a table (ha!) of what SQL and relational algebar operations correspond to
what Octave @code{table} operations.

In this table, @code{t} is a variable holding a @code{table} array, and @code{ix} is 
some indexing expression.

@multitable @columnfractions .33 .33 .33
@headitem SQL @tab Relational @tab Octave @code{table}
@item SELECT
@tab PROJECT
@tab @code{subsetvars}, @code{t(:,ix)}
@item WHERE
@tab RESTRICT
@tab @code{subsetrows}, @code{t(ix,:)}
@item INNER JOIN
@tab JOIN
@tab @code{innerjoin}
@item OUTER JOIN
@tab OUTER JOIN
@tab @code{outerjoin}
@item FROM @code{@var{table1}, @var{table2}, @dots{}}
@tab Cartesian product
@tab @code{cartesian}
@item GROUP BY
@tab SUMMARIZE
@tab @code{groupby}
@item DISTINCT
@tab (automatic)
@tab @code{unique(t)}
@end multitable

Note that there is one big difference between relational algebra and SQL & Octave
@code{table}: Relations in relational algebra are sets, not lists.
There are no duplicate rows in relational algebra, and there is no ordering.
So every operation there does an implicit @code{DISTINCT}/@code{unique()} on its
results, and there‘s no @code{ORDER BY}/@code{sort()}. This is not the case in SQL
or Octave @code{table}.

Note for users coming from Matlab: Matlab does not provide a general @code{groupby}
function. Instead, you have to variously use @code{rowfun}, @code{grpstats},
@code{groupsummary}, and manual code to accomplish “group by” operations.

Note: I wrote this based on my understanding of relational algebra from reading
C. J. Date books. Other people’s understanding and terminology may differ. - apjanke



@node Date and Time Representation
@chapter Date and Time Representation

Tablicious provides the @code{datetime} class for representing points in time.

There’s also @code{duration} and @code{calendarDuration} for representing
periods or durations of time. Like vector quantities along the time line,
as opposed to @code{datetime} being a point along the time line.

@node datetime Class
@section @code{datetime} Class

A @code{datetime} is an array object that represents points in time in the familiar
Gregorian calendar.

This is an attempt to reproduce the functionality of Matlab's @code{datetime}. It
also contains some Octave-specific extensions.

The underlying representation is that of a datenum (a @code{double}
containing the number of days since the Matlab epoch), but encapsulating it in an
object provides several benefits: friendly human-readable display, type safety,
automatic type conversion, and time zone support. In addition to the underlying
datenum array, a @code{datetime} inclues an optional @code{TimeZone} property 
indicating what time zone the datetimes are in.

So, basically, a @code{datetime} is an object wrapper around a datenum array,
plus time zone support.

@menu
* Datenum Compatibility::
@end menu

@node Datenum Compatibility
@subsection Datenum Compatibility

While the underlying data representation of @code{datetime} is compatible with
(in fact, identical to) that of datenums, you cannot directly combine them
via assignment, concatenation, or most arithmetic operations.

This is because of the signature of the @code{datetime} constructor. When combining
objects and primitive types like @code{double}, the primitive type is promoted to an
object by calling the other object's one-argument constructor on it. However, the
one-argument numeric-input consstructor for @code{datetime} does not accept datenums:
it interprets its input as datevecs instead. This is due to a design decision on
Matlab's part; for compatibility, Octave does not alter that interface.

To combine @code{datetime}s with datenums, you can convert the datenums to @code{datetime}s
by calling @code{datetime.ofDatenum} or @code{datetime(x, 'ConvertFrom', 'datenum')}, or you
can convert the @code{datetime}s to datenums by accessing its @code{dnums} field with
@code{x.dnums}.

Examples:

@example
dt = datetime('2011-03-04')
dn = datenum('2017-01-01')
[dt dn]
    @result{} error: datenum: expected date vector containing [YEAR, MONTH, DAY, HOUR, MINUTE, SECOND]
[dt datetime.ofDatenum(dn)]
    @result{} 04-Mar-2011   01-Jan-2017
@end example

Also, if you have a zoned @code{datetime}, you can’t combine it with a datenum, because datenums
do not carry time zone information.


@node Time Zones
@section Time Zones

Tablicious has support for representing dates in time zones and for converting between time zones.

A @code{datetime} may be "zoned" or "zoneless". A zoneless @code{datetime} does not have a time zone
associated with it. This is represented by an empty @code{TimeZone} property on the @code{datetime}
object. A zoneless @code{datetime} represents the local time in some unknown time zone, and assumes a
continuous time scale (no DST shifts).

A zoned @code{datetime} is associated with a time zone. It is represented by having the time zone's
IANA zone identifier (e.g. @code{'UTC'} or @code{'America/New_York'}) in its @code{TimeZone} 
property. A zoned @code{datetime} represents the local time in that time zone.

By default, the @code{datetime} constructor creates unzoned @code{datetime}s. To 
make a zoned @code{datetime}, either pass the @code{'TimeZone'} option to the constructor, 
or set the @code{TimeZone} property after object creation. Setting the @code{TimeZone}
property on a zoneless @code{datetime} declares that it's a local time in that time zone.
Setting the @code{TimeZone} property on a zoned @code{datetime} turns it back into a 
zoneless @code{datetime} without changing the local time it represents.

You can tell a zoned from a zoneless time zone in the object display because the time zone
is included for zoned @code{datetime}s.

@example
% Create an unzoned datetime
d = datetime('2011-03-04 06:00:00')
    @result{}  04-Mar-2011 06:00:00

% Create a zoned datetime
d_ny = datetime('2011-03-04 06:00:00', 'TimeZone', 'America/New_York')
    @result{}  04-Mar-2011 06:00:00 America/New_York
% This is equivalent
d_ny = datetime('2011-03-04 06:00:00');
d_ny.TimeZone = 'America/New_York'
    @result{}  04-Mar-2011 06:00:00 America/New_York

% Convert it to Chicago time
d_chi.TimeZone = 'America/Chicago'
    @result{}  04-Mar-2011 05:00:00 America/Chicago
@end example

When you combine two zoned @code{datetime}s via concatenation, assignment, or 
arithmetic, if their time zones differ, they are converted to the time zone of
the left-hand input.

@example
d_ny = datetime('2011-03-04 06:00:00', 'TimeZone', 'America/New_York')
d_la = datetime('2011-03-04 06:00:00', 'TimeZone', 'America/Los_Angeles')
d_la - d_ny
    @result{} 03:00:00
@end example

You cannot combine a zoned and an unzoned @code{datetime}. This results in an error
being raised.

@quotation Warning
Normalization of "nonexistent" times (like between 02:00 and 03:00 on a "spring forward"
DST change day) is not implemented yet. The results of converting a zoneless local time
into a time zone where that local time did not exist are currently undefined.
@end quotation

@menu
* Defined Time Zones::
@end menu

@node Defined Time Zones
@subsection Defined Time Zones

Tablicious's time zone data is drawn from the @uref{https://www.iana.org/time-zones, 
IANA Time Zone Database}, also known as the “Olson Database”. Tablicious includes a
copy of this database in its distribution so it can work on Windows, which does
not supply it like Unix systems do.

You can use the @code{timezones} function to list the time zones known to Tablicious. These will be
all the time zones in the IANA database on your system (for Linux and macOS) or in the IANA
time zone database redistributed with Tablicious (for Windows).

@quotation Note
The IANA Time Zone Database only covers dates from about the year 1880 to 2038. Converting
time zones for @code{datetime}s outside that range is currently unimplemented. (Tablicious 
needs to add support for proleptic POSIX time zone rules, which are used to govern
behavior outside that date range.)
@end quotation

@node Durations
@section Durations

@menu
* duration Class::
* calendarDuration Class::
@end menu

@node duration Class
@subsection @code{duration} Class

A @code{duration} represents a period of time in fixed-length seconds (or minutes, hours,
or whatever you want to measure it in.)

A @code{duration} has a resolution of about a nanosecond for typical dates. The underlying
representation is a @code{double} representing the number of days elapsed, similar to a 
datenum, except it's interpreted as relative to some other reference point you provide,
instead of being relative to the Matlab/Octave epoch.

You can add or subtract a @code{duration} to a @code{datetime} to get another @code{datetime}.
You can also add or subtract @code{durations} to each other.

@node calendarDuration Class
@subsection @code{calendarDuration} Class

A @code{calendarDuration} represents a period of time in variable-length calendar 
components. For example, years and months can have varying numbers of days, and days
in time zones with Daylight Saving Time have varying numbers of hours. A 
@code{calendarDuration} does arithmetic with "whole" calendar periods.

@code{calendarDuration}s and @code{duration}s cannot be directly combined, because
they are not semantically equivalent. (This may be relaxed in the future to allow
@code{duration}s to be interpreted as numbers of days when combined with
@code{calendarDuration}s.)

@example
d = datetime('2011-03-04 00:00:00')
    @result{} 04-Mar-2011
cdur = calendarDuration(1, 3, 0)
    @result{} 1y 3mo
d2 = d + cdur
    @result{} 04-Jun-2012
@end example


@node Validation Functions
@chapter Validation Functions

Tablicious provides several validation functions which can be used to check properties
of function arguments, variables, object properties, and other expressions. These can
be used to express invariants in your program and catch problems due to input errors,
incorrect function usage, or other bugs.

These validation functions are named following the pattern @code{mustBeXxx}, where @code{Xxx}
is some property of the input it is testing. Validation functions may check the type,
size, or other aspects of their inputs.

The most common place for validation functions to be used will probably be at the
beginning of functions, to check the input arguments and ensure that the contract of
the function is not being violated. If in the future Octave gains the ability to
declaratively express object property constraints, they will also be of use there.

Be careful not to get too aggressive with the use of validation functions: while using
them can make sure invariants are followed and your program is correct, they also reduce
the code’s ability to make use of duck typing, reducing its flexibility. Whether you want
to make this trade-off is a design decision you will have to consider.

When a validation function’s condition is violated, it raises an error that includes a
description of the violation in the error message. This message will include a label for
the input that describes what is being tested. By default, this label is initialized
with @code{inputname()}, so when you are calling a validator on a function argument or
variable, you will generally not need to supply a label. But if you’re calling it on
an object property or an expression more complex than a simple variable reference, the
validator cannot automatically detect the input name for use in the label. In this case,
make use of the optional trailing argument(s) to the functions to manually supply a
label for the value being tested.

@example
% Validation of a simple variable does not need a label
mustBeScalar (x);
% Validation of a field or property reference does need a label
mustBeScalar (this.foo, 'this.foo');
@end example


@node Example Data Sets
@chapter Example Data Sets

Tablicious comes with several example data sets that you can use to explore how
its functions and objects work. These are accessed through the
@code{octave.datasets} and @code{octave.dataset} classes.

To see a list of the available data sets, run @code{octave.datasets.list()}.
Then to load one of the example data sets, run
@code{octave.datasets.load('examplename')}. For example:

@example
octave.datasets.list
t = octave.datasets.load('cupcake')
@end example

You can also load it by calling @code{octave.dataset.<name>}. This does
the same thing. For example:

@example
t = octave.dataset.cupcake
@end example

When you load a data set, it either returns all its data in a single variable
(if you capture it), or loads its data into one or more variables in your
workspace (if you call it with no outputs).

Each example data set comes with help text that describes the data set and
provides examples of how to work with it. This help is found using the @code{doc}
command on @code{octave.dataset.<name>}, where @var{<name>} is the name of
the data set.

For example:

@example
doc octave.dataset.cupcake
@end example

(The command @code{help octave.dataset.<name>} ought to work too, but it
currently doesn’t. This may be due to an issue with Octave’s @code{help}
command.)

@node Data Sets from R
@section Data Sets from R

Many of Tablicious’ example data sets are based on the example datasets
found in R’s @code{datasets} package. R can be found at
@url{https://www.r-project.org/}, and documentation for its @code{datasets}
is at @url{https://rdrr.io/r/datasets/datasets-package.html}.
Thanks to the R developers for producing the original data sets here.

Tablicious’ examples’ code tries to replicate the R examples, so it can
be useful to compare the two of them if you are moving from one language to
another.

Core Octave currently lacks some of the plotting features found in the R
examples, such as LOWESS smoothing and linear model characteristic plots, so
you will just find “TODO” placeholders for these in Tablicious’ example code.

@node Missing Functionality
@chapter Missing Functionality

Tablicious is based on Matlab’s table and date/time APIs and supports most of
their major functionality.
But not all of it is implemented yet. The missing parts are currently:

@itemize
@item File I/O like @code{readtable()} and @code{writetable()}
@item @code{summary()} @code{categorical}
@item Assignment to table variables using @code{.}-indexing
@item @code{timetable}
@item POSIX time zone support for years outside the IANA time zone database coverage
@item Week-of-year (ISO calendar) calculations
@item Various @code{'ConvertFrom'} forms for @code{datetime} and @code{duration}
@item Support for LDML formatting for @code{datetime}
@item Various functions:
@itemize
@item @code{between}
@item @code{caldiff}
@item @code{dateshift}
@item @code{week}
@end itemize
@item @code{isdst}, @code{isweekend}
@item @code{calendarDuration.split}
@item @code{duration.Format} support
@item Moving window methods in @code{fillmissing}
@item @code{UTCOffset} and @code{DSTOffset} fields in the output of @code{timezones()}
@item Plotting support
@end itemize

It is the author’s hope that all these will be implemented some day.

These areas of missing functionality are all tracked on the Tablicious issue
tracker at @url{https://github.com/apjanke/octave-tablicious/issues} and
@url{https://github.com/users/apjanke/projects/1}.


@node API Reference
@chapter API Reference

@REFERENCE_SECTION(Tablicious)



@node Copying
@chapter Copying


@section Package Copyright

Tablicious for Octave is covered by the GNU GPLv3.

All the code in the package is GNU GPLv3.

The Fisher Iris dataset is Public Domain.

@section Manual Copyright

@insertcopying

@bye
