# Changes from M11 to M12

## Applications

- Rainbow

  - Fixed YAML escaping/un-escaping for quotes in the Edit Code Finder Rules tool.
  - Added the Create New Document command.

- Tikal

  - Updated the `-tt` option to make the server parameters optional and default to the Amagama server.

## Filters

- PO Filter

  - Implemented case of no quote on msgid/msgstr/etc. line.
  - Improved error reporting with line numbers.
  - Added msgctxt in the generated resname values. This allows two entries with the same source and same domain but with a different context to have distinct resname auto-generated values.

- MIF Filter

  - Implemented extraction for index markers (`default=yes`) and hyperlinks (`default=no`).
  - When needed, leading codes (font, etc.) are now extracted as inline code rather than kept in the skeleton.

- Regex Filter

  - Added support for escaped character notation that uses doubled characters (e.g. `""`).
  - Made the back-slash escape notation optional.

- XML Filter

  - Fixed bug where standalone=“yes” was re-written standalone=“true” in the XML declaration. This fixes issue #173.

- IDML Filter

  - Improved support for special characters (hair spaces, forced line-breaks, etc.)

## Steps

- Added the Remove Target Step

  - to remove target entries from text units.

- Added the Inline Codes Simplifier Step

  - to join adjacent inline codes in text units.

- Added the GTT Batch Translation Step

  - to create TM files using Google Translator Toolkit.

- Added the Repetition Analysis Step

  - to detect text repetitions for word count report.

- Added the Extraction Verification Step

  - to check if a document can be extracted/merged/re-extracted and generate the same events (this help verifying that extracted text can be safely merged back).

- Rainbow Translation Kit Creation Step

  - Added option for creating XLIFF 2.0 packages. This is for experimental tests only. XLIFF 2.0 is not defined yet.
  - Added the option of setting the sentence_seg flag in the project for OmegaT packages.
  - Fixed synchronization error with empty source on merging PO and Transifex packages.
  - Implemented support for input files without filter configuration: they are copied into the original folder if the selected kit has one.

- Rainbow Translation Kit Merging Step

  - Fixed merging for segmented entries.
  - Added option to preserve or not segmentation for the next step (false to save time if there is no step afterward)

- Leveraging Step

  - Added the option to add a prefix to the leveraged translation, when a given threshold is equal or below the score of the leveraged match, and depending on the target content.
  - Added option to copy the source when the source content has no text (but may have white-spaces and/or codes).

- Search and Replace Step

  - Added the option “replace all instances of the pattern” to replace all or only the first match in each item searched.

- Quality Check Step

  - Changed the inline code verification so only selected types of codes are ignored rater than all the ones without nativae data. now, by default only `mrk` and `x-df-s` types are ignored.

- Segmentation Step

  - Prevented the options to be applied when no segmentation is done.

- Line-Break Conversion Step

  - Updated output so the output can be the same file as the input.

- Byte-Order-Mark Conversion Step

  - Updated output so the output can be the same file as the input.

- RTF Conversion Step

  - Updated output so the output can be the same file as the input.

- XML Character Fixing Step

  - Updated output so the output can be the same file as the input.

- Encoding Conversion Step

  - Updated output so the output can be the same file as the input.

- Term Extraction Step

  - Changed code to include numbers as part of the possible terms.

## Connectors

- GlobalSight Connector

  - Changed the conversion of the score to handle the floats sent by the latest API, instead of integers.

- Translate Toolkit TM Connector

  - Updated the default to use the Amagama server.
  - Added support for letter-coded inline codes.

- Removed Cross-Language Gateway MT services connector from default distribution.

- Microsoft MT Connector

  - Implemented support for inline codes.

## Libraries

- Fixed corrupted `DefaultFilters.properties` so XINI filter and MIF Filter UI are accessible again.
- Added support for encoded storage of strings (e.g. passwords) in several types of parameters files.
- Added output helper methods to RawDocument.
- The Lucene library has been updated to 3.1.0. You may need to re-index your Pensieve TMs. You can do this by exporting the TM to TMX, then re-importing it back.
- Updated `TextUnit` to use `ITextUnit` and a new implementation. This is a major low-level refactoring that comes with some behavior changes and additions:
  - in `createTarget()` `IResource.COPY_CONTENT` should be replaced by `IResource.COPY_SEGMENTED_CONTENT` to have the same behavior.
  - the new `getTargetSegments()` does create a target if it does not exists (with a `COPY_SEGMENTATION` option).
- Moved `IQuery` and `QueryResult` from the `okapi-lib-translation` project to the core (`common.query` package).
- Implemented plugin support for connectors.