ANSI Compliance

In Spark SQL, there are two options to comply with the SQL standard: spark.sql.ansi.enabled and spark.sql.storeAssignmentPolicy (See a table below for details).

When spark.sql.ansi.enabled is set to true, Spark SQL uses an ANSI compliant dialect instead of being Hive compliant. For example, Spark will throw an exception at runtime instead of returning null results if the inputs to a SQL operator/function are invalid. Some ANSI dialect features may be not from the ANSI SQL standard directly, but their behaviors align with ANSI SQL’s style.

Moreover, Spark SQL has an independent option to control implicit casting behaviours when inserting rows in a table. The casting behaviours are defined as store assignment rules in the standard.

When spark.sql.storeAssignmentPolicy is set to ANSI, Spark SQL complies with the ANSI store assignment rules. This is a separate configuration because its default value is ANSI, while the configuration spark.sql.ansi.enabled is disabled by default.

The following subsections present behaviour changes in arithmetic operations, type conversions, and SQL parsing when the ANSI mode enabled. For type conversions in Spark SQL, there are three kinds of them and this article will introduce them one by one: cast, store assignment and type coercion.

Arithmetic Operations

In Spark SQL, arithmetic operations performed on numeric types (with the exception of decimal) are not checked for overflows by default. This means that in case an operation causes overflows, the result is the same with the corresponding operation in a Java/Scala program (e.g., if the sum of 2 integers is higher than the maximum value representable, the result is a negative number). On the other hand, Spark SQL returns null for decimal overflows. When spark.sql.ansi.enabled is set to true and an overflow occurs in numeric and interval arithmetic operations, it throws an arithmetic exception at runtime.

-- `spark.sql.ansi.enabled=true`
SELECT 2147483647 + 1;
java.lang.ArithmeticException: integer overflow

SELECT abs(-2147483648);
java.lang.ArithmeticException: integer overflow

-- `spark.sql.ansi.enabled=false`
SELECT 2147483647 + 1;
+----------------+
|(2147483647 + 1)|
+----------------+
|     -2147483648|
+----------------+

SELECT abs(-2147483648);
+----------------+
|abs(-2147483648)|
+----------------+
|     -2147483648|
+----------------+

Cast

When spark.sql.ansi.enabled is set to true, explicit casting by CAST syntax throws a runtime exception for illegal cast patterns defined in the standard, e.g. casts from a string to an integer.

The CAST clause of Spark ANSI mode follows the syntax rules of section 6.13 “cast specification” in ISO/IEC 9075-2:2011 Information technology — Database languages - SQL — Part 2: Foundation (SQL/Foundation), except it specially allows the following straightforward type conversions which are disallowed as per the ANSI standard:

• NumericType <=> BooleanType
• StringType <=> BinaryType
• ArrayType => String
• MapType => String
• StructType => String

The valid combinations of source and target data type in a CAST expression are given by the following table. “Y” indicates that the combination is syntactically valid without restriction and “N” indicates that the combination is not valid.

In the table above, all the CASTs that can cause runtime exceptions are marked as red Y:

• CAST(Numeric AS Numeric): raise an overflow exception if the value is out of the target data type’s range.
• CAST(String AS (Numeric/Date/Timestamp/Interval/Boolean)): raise a runtime exception if the value can’t be parsed as the target data type.
• CAST(Array AS Array): raise an exception if there is any on the conversion of the elements.
• CAST(Map AS Map): raise an exception if there is any on the conversion of the keys and the values.
• CAST(Struct AS Struct): raise an exception if there is any on the conversion of the struct fields.

-- Examples of explicit casting

-- `spark.sql.ansi.enabled=true`
SELECT CAST('a' AS INT);
java.lang.NumberFormatException: invalid input syntax for type numeric: a

SELECT CAST(2147483648L AS INT);
java.lang.ArithmeticException: Casting 2147483648 to int causes overflow

SELECT CAST(DATE'2020-01-01' AS INT)
org.apache.spark.sql.AnalysisException: cannot resolve 'CAST(DATE '2020-01-01' AS INT)' due to data type mismatch: cannot cast date to int.
To convert values from date to int, you can use function UNIX_DATE instead.

-- `spark.sql.ansi.enabled=false` (This is a default behaviour)
SELECT CAST('a' AS INT);
+--------------+
|CAST(a AS INT)|
+--------------+
|          null|
+--------------+

SELECT CAST(2147483648L AS INT);
+-----------------------+
|CAST(2147483648 AS INT)|
+-----------------------+
|            -2147483648|
+-----------------------+

SELECT CAST(DATE'2020-01-01' AS INT)
+------------------------------+
|CAST(DATE '2020-01-01' AS INT)|
+------------------------------+
|                          null|
+------------------------------+

-- Examples of store assignment rules
CREATE TABLE t (v INT);

-- `spark.sql.storeAssignmentPolicy=ANSI`
INSERT INTO t VALUES ('1');
org.apache.spark.sql.AnalysisException: Cannot write incompatible data to table '`default`.`t`':
- Cannot safely cast 'v': string to int;

-- `spark.sql.storeAssignmentPolicy=LEGACY` (This is a legacy behaviour until Spark 2.x)
INSERT INTO t VALUES ('1');
SELECT * FROM t;
+---+
|  v|
+---+
|  1|
+---+

Store assignment

As mentioned at the beginning, when spark.sql.storeAssignmentPolicy is set to ANSI(which is the default value), Spark SQL complies with the ANSI store assignment rules on table insertions. The valid combinations of source and target data type in table insertions are given by the following table.

* Spark doesn’t support interval type table column.

** For Array/Map/Struct types, the data type check rule applies recursively to its component elements.

During table insertion, Spark will throw exception on numeric value overflow.

CREATE TABLE test(i INT);

INSERT INTO test VALUES (2147483648L);
java.lang.ArithmeticException: Casting 2147483648 to int causes overflow

Type coercion

Type Promotion and Precedence

When spark.sql.ansi.enabled is set to true, Spark SQL uses several rules that govern how conflicts between data types are resolved. At the heart of this conflict resolution is the Type Precedence List which defines whether values of a given data type can be promoted to another data type implicitly.

* For least common type resolution float is skipped to avoid loss of precision.

** For a complex type, the precedence rule applies recursively to its component elements.

Special rules apply for string literals and untyped NULL. A NULL can be promoted to any other type, while a string literal can be promoted to any simple data type.

This is a graphical depiction of the precedence list as a directed tree:

Least Common Type Resolution

The least common type from a set of types is the narrowest type reachable from the precedence list by all elements of the set of types.

The least common type resolution is used to:

• Decide whether a function expecting a parameter of a type can be invoked using an argument of a narrower type.
• Derive the argument type for functions which expect a shared argument type for multiple parameters, such as coalesce, least, or greatest.
• Derive the operand types for operators such as arithmetic operations or comparisons.
• Derive the result type for expressions such as the case expression.
• Derive the element, key, or value types for array and map constructors. Special rules are applied if the least common type resolves to FLOAT. With float type values, if any of the types is INT, BIGINT, or DECIMAL the least common type is pushed to DOUBLE to avoid potential loss of digits.

-- The coalesce function accepts any set of argument types as long as they share a least common type. 
-- The result type is the least common type of the arguments.
> SET spark.sql.ansi.enabled=true;
> SELECT typeof(coalesce(1Y, 1L, NULL));
BIGINT
> SELECT typeof(coalesce(1, DATE'2020-01-01'));
Error: Incompatible types [INT, DATE]

> SELECT typeof(coalesce(ARRAY(1Y), ARRAY(1L)));
ARRAY<BIGINT>
> SELECT typeof(coalesce(1, 1F));
DOUBLE
> SELECT typeof(coalesce(1L, 1F));
DOUBLE
> SELECT (typeof(coalesce(1BD, 1F)));
DOUBLE

-- The substring function expects arguments of type INT for the start and length parameters.
> SELECT substring('hello', 1Y, 2);
he
> SELECT substring('hello', '1', 2);
he
> SELECT substring('hello', 1L, 2);
Error: Argument 2 requires an INT type.
> SELECT substring('hello', str, 2) FROM VALUES(CAST('1' AS STRING)) AS T(str);
Error: Argument 2 requires an INT type.

SQL Functions

The behavior of some SQL functions can be different under ANSI mode (spark.sql.ansi.enabled=true).

• size: This function returns null for null input.
• element_at:
  • This function throws ArrayIndexOutOfBoundsException if using invalid indices.
  • This function throws NoSuchElementException if key does not exist in map.
• elt: This function throws ArrayIndexOutOfBoundsException if using invalid indices.
• parse_url: This function throws IllegalArgumentException if an input string is not a valid url.
• to_date: This function should fail with an exception if the input string can’t be parsed, or the pattern string is invalid.
• to_timestamp: This function should fail with an exception if the input string can’t be parsed, or the pattern string is invalid.
• unix_timestamp: This function should fail with an exception if the input string can’t be parsed, or the pattern string is invalid.
• to_unix_timestamp: This function should fail with an exception if the input string can’t be parsed, or the pattern string is invalid.
• make_date: This function should fail with an exception if the result date is invalid.
• make_timestamp: This function should fail with an exception if the result timestamp is invalid.
• make_interval: This function should fail with an exception if the result interval is invalid.
• next_day: This function throws IllegalArgumentException if input is not a valid day of week.

SQL Operators

The behavior of some SQL operators can be different under ANSI mode (spark.sql.ansi.enabled=true).

• array_col[index]: This operator throws ArrayIndexOutOfBoundsException if using invalid indices.
• map_col[key]: This operator throws NoSuchElementException if key does not exist in map.

Useful Functions for ANSI Mode

When ANSI mode is on, it throws exceptions for invalid operations. You can use the following SQL functions to suppress such exceptions.

• try_cast: identical to CAST, except that it returns NULL result instead of throwing an exception on runtime error.
• try_add: identical to the add operator +, except that it returns NULL result instead of throwing an exception on integral value overflow.
• try_divide: identical to the division operator /, except that it returns NULL result instead of throwing an exception on dividing 0.

SQL Keywords

When spark.sql.ansi.enabled is true, Spark SQL will use the ANSI mode parser. In this mode, Spark SQL has two kinds of keywords:

• Reserved keywords: Keywords that are reserved and can’t be used as identifiers for table, view, column, function, alias, etc.
• Non-reserved keywords: Keywords that have a special meaning only in particular contexts and can be used as identifiers in other contexts. For example, EXPLAIN SELECT ... is a command, but EXPLAIN can be used as identifiers in other places.

When the ANSI mode is disabled, Spark SQL has two kinds of keywords:

• Non-reserved keywords: Same definition as the one when the ANSI mode enabled.
• Strict-non-reserved keywords: A strict version of non-reserved keywords, which can not be used as table alias.

By default spark.sql.ansi.enabled is false.

Below is a list of all the keywords in Spark SQL.