# Intel® Integrated Performance Primitives Examples

# Disclaimer

## Legal Information

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.
A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.

Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined." Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.
The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.
Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.
Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm

Intel processor numbers are not a measure of performance. Processor numbers differentiate features within each processor family, not across different processor families. Go to: http://www.intel.com/products/processor_number/

BlueMoon, BunnyPeople, Celeron, Celeron Inside, Centrino, Centrino Inside, Core Inside, i960, Intel, the Intel logo, Intel Atom, Intel Atom Inside, Intel Core, Intel Inside, Intel Inside logo, Intel NetBurst, Intel NetMerge, Intel NetStructure, Intel SingleDriver, Intel SpeedStep, Intel Sponsors of Tomorrow., the Intel Sponsors of Tomorrow. logo, Intel StrataFlash, Intel Viiv, Intel vPro, Intel XScale, InTru, the InTru logo, InTru soundmark, Itanium, Itanium Inside, MCS, MMX, Moblin, Pentium, Pentium Inside, skoool, the skoool logo, Sound Mark, The Journey Inside, vPro Inside, VTune, Xeon, and Xeon Inside are trademarks of Intel Corporation in the U.S. and other countries.
* Other names and brands may be claimed as the property of others.

Microsoft, Windows, Visual Studio, Visual C++, and the Windows logo are trademarks, or registered trademarks of Microsoft Corporation in the United States and/or other countries.

Java is a registered trademark of Oracle and/or its affiliates.

©Intel Corporation. All rights reserved.

Optimization Notice
Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice revision #20110804

# License Definitions

By downloading and installing this product, you hereby agree that the accompanying materials are being provided to you under the terms and conditions of the End User License Agreement for the Intel® Integrated Performance Primitives (Intel® IPP) product previously accepted by you.

# System Requirements

Hardware requirements:

- A system based on Intel® Pentium® processor, Intel® Xeon® processor, or subsequent IA-32 architecture based system;

- A system based on a processor with Intel® 64 architecture.

Software requirements:

- For the basic system requirements on operating system, Intel® IPP library, IDE and compiler, refer to the corresponding Release Notes document.

For more information please see https://software.intel.com/en-us/ipp [Intel® Integrated Performance Primitives].

# What’s New

The following updates have been made to Intel® IPP functions for Zlib:

- Removed source code patch file for Zlib version 1.2.5.3. If you want to speed up this particular version of Zlib, please use patch file from previous distributions of Intel® IPP

- Significantly improved performance of the data compression functions without degradation of the compression ratio

- Implemented a new fastest data compression mode. The compression ratio of the new mode is a bit less than the fast mode’s ratio - compression level 1, but performance is significantly better. To use this mode, specify the negative compression level equal to -2 (since -1 is used for default compression level in Zlib) when calling the `deflateInit` function:

        deflateInit(&deflate_stream, -2);

You can use symbolic name `Z_IPP_FAST_COMPRESSION`, which is equal to -2:

        deflateInit(&deflate_stream, Z_IPP_FAST_COMPRESSION);

- Introduced new sub-ranges of compression levels from 11 to 29

  - The standard levels 1..9 are compatible with original Zlib

  - Levels 11..19 are similar to 1..9, but fit better when compressing large files (more than 1 MB)

  - Levels 21..29 are for highly compressible files with compression ratio equal to 30x and higher.

- Activated Zlib `deflateTune` function. The following is quote from Zlib programming manual:

    ZEXTERN int ZEXPORT deflateTune OF((z_streamp strm,
                                        int good_length,
                                        int max_lazy,
                                        int nice_length,
                                        int max_chain));
    Fine tune deflate's internal compression parameters. This should only be used by
    someone who understands the algorithm used by Zlib's +deflate+ for searching for the
    best matching string, and even then only by the most fanatic optimizer trying
    to squeeze out the last compressed bit for their specific input data.

    Read the deflate.c source code for the meaning of the max_lazy, good_length,
    nice_length, and max_chain parameters.

The described function arguments now affect Intel® IPP data compression functions.

You can find the default sets of string matching parameters in the `configuration_table` array inside `deflate.c` source code file.

- Added the new "trained deflate tables" feature for `Z_IPP_FAST_COMPRESSION` - fastest compression mode (see New Intel® IPP Functions for Compression with Trained Deflate Tables).

# How to Build Zlib with Intel® Integrated Performance Primitives Library

The data compression domain of the Intel® IPP library contains several functions that can speed up the well-known Zlib library (http://zlib.net) in both data compression and decompression operations. For the detailed description of these functions, refer to the Volume 1: "Signal Processing/Data Compression Functions/Dictionary-Based Compression Functions/ZLIB Coding Functions" section of the Intel® IPP Developer Reference.

To use these functions, before building the library you need to replace the original source code of several Zlib functions with Intel® IPP functions calls.

This document provides instructions on how to prepare the Zlib source code and build the library.

## Prerequisites

The Zlib repository contains several stable versions of the library: v.1.2.6.1, v.1.2.7.3, v.1.2.8 and v.1.2.11. You may want to update and speed up a particular Zlib version, which is used in your application, or on your operating system. For that, Intel® IPP product provides four source code patch files applicable to specific versions of Zlib.

## Downloading, Patching and Build on Linux* and macOS*

### Downloading Zlib Source Code

Before building the library, you need to download Zlib source code files from the Zlib site (http://zlib.net). The following table provides links to specific Zlib versions that can be updated with Intel® IPP library function calls.

Zlib version | Web address
--- | ---
`1.2.6.1` | http://zlib.net/fossils/zlib-1.2.6.1.tar.gz
`1.2.7.3` | http://zlib.net/fossils/zlib-1.2.7.3.tar.gz
`1.2.8` | http://zlib.net/fossils/zlib-1.2.8.tar.gz
`1.2.11` | http://zlib.net/zlib-1.2.11.tar.gz

Download the tar file from the specified address to a working directory. Unpack the downloaded archive with the following command on Linux*/macOS* systems.

    $ tar xvf <archive file name>

This command will create the `zlib-<version>` folder in your working directory, where `<version>` is selected Zlib version number from 1.2.6.1 to 1.2.11.

### Patching the Source Code

1. Change directory to the `zlib-<version>` folder.

2. Apply the corresponding source code patch file `zlib-<version>.patch.bin` from Intel® IPP product distribution with the following command

    $ patch -p1 < "path to corresponding patch file"

For example, to patch Zlib v.1.2.6.1, go to `zlib-1.2.6.1` directory and run the following command

    $ patch -p1 < ../zlib-1.2.6.1.patch.bin

If the patching process completed successfully, you will see the following messages:

    patching file adler32.c
    patching file crc32.c
    patching file deflate.c
    patching file inflate.c
    patching file inftrees.h
    patching file trees.c

If something goes wrong, for example, if the `adler32.c` file has been changed (nonoriginal), you will see the diagnostics messages:

    patching file adler32.c
    Hunk #1 FAILED at 5.
    patching file crc32.c
    patching file deflate.c
    patching file inflate.c
    patching file inftrees.h
    patching file trees.c

### Configuring and Building Zlib Library

After Zlib source code files are patched successfully, you need to create a makefile for building process. Basically, you need to

- Add the `WITH_IPP` definition to compiler command line

- Add the `$IPPROOT/include` directory to the list of header files search for compiler

- Add Intel® IPP libraries to the list of linker input files.

On Linux* OS, you can implement the above steps with the standard Zlib configure/make sequence and the following commands (32-bit example):

    # Build dynamic Zlib library with static Intel(R) IPP linkage on Linux*
    $ export CFLAGS="-m32 -DWITH_IPP -I$IPPROOT/include"
    $ export LDFLAGS="$IPPROOT/lib/ia32/libippdc.a $IPPROOT/lib/ia32/libipps.a $IPPROOT/lib/ia32/libippcore.a"
    $ ./configure
    $ make shared

Here, the `-m32` option specifies IA-32 target CPU architecture. For Intel® 64 architecture the compiler option should be `-m64`.

The above commands assume that `IPPROOT` environment variable points to head of Intel® IPP location.

Configure your Intel® IPP working environment according to "Get Started with Intel® Integrated Performance Primitives" document.

Note | The conditional compilation statements, used for Zlib source code files modification, allow to build original version of Zlib from the same source code files. For that, you need to build Zlib library without modification of `CFLAGS` and `LDFLAGS`. If `WITH_IPP` definition doesn’t exist in compiler options, Zlib source code files are compiled as unpatched.

### Static and Dynamic Linking to Intel® IPP Libraries

When building the `libz` dynamic shared object, you can use either static or dynamic linking to Intel® IPP libraries. Different linking modes have their advantages and disadvantages. For example, static linking to Intel® IPP provides target object (`libz.so`, or `libz.dylib` in our case) independence from environment, easy transfer from one computer to another. On the other hand, statically linked object has bigger size and can’t benefit from newer (updated) version of Intel® IPP without re-linking.

The `libz` dynamic libraries, linked statically with Intel® IPP, are self-sufficient. It means that when you build an application using these libraries, you don’t need to add Intel® IPP libraries to the linker command line (unless your application uses Intel® IPP apart from `libz`). All references to Intel® IPP functions are already resolved, and required Intel® IPP functions code is inside the `libz` dynamic library.

#### Build Zlib with Static Linking to Intel® IPP

On Linux* OS, when the below linker flags are specified by the `LDFLAGS` environment variable, Zlib is linked to Intel® IPP libraries (.a files) statically:

    $ export LDFLAGS="$IPPROOT/lib/intel64/libippdc.a $IPPROOT/lib/intel64/libipps.a \
                      $IPPROOT/lib/intel64/libippcore.a"

You can check the independence of the resulting dynamic library `libz.so` from Intel® IPP libraries with the "ldd" tool.

For example,

    $ ldd libz.so
            linux-vdso.so.1 =>  (0x00007fff885fe000)
            libc.so.6 => /lib64/libc.so.6 (0x00007fb169e38000)
            /lib64/ld-linux-x86-64.so.2 (0x00007fb16a503000)

You can see that the `libz.so` dynamic library depends on system dynamic libraries only. The size of the resulting `libz` library in this case will be more than 700 KBytes because it contains optimizations for all CPUs supported by Intel® IPP.

On macOS*, the similar tool is called "otool". For example,

    $ otool -L libz.dylib
    libz.dylib:
            /usr/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
            /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

You can use the following command to build an application with such kind of library:

    $ cc -o zlib_test -Izlib/include zlib_test.c -L. -lz

Here, `zlib/include` is a directory where zlib header files reside (in most cases `zlib.h` and `zconf.h` are required), "-L." means that Linux* linker will be searching next libraries in the current directory, "-lz" means that the linker will look for the `libz.so` file (shared linking is default method on most Linux* systems).

To run built application on Linux* system, you need to modify the `LD_LIBRARY_PATH` environment variable, which describes the search path for dynamic libraries:

    $ export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH
    $ ./zlib_test

On macOS* systems the corresponding environment variable is `DYLD_LIBRARY_PATH`:

    $ export DYLD_LIBRARY_PATH=.:$DYLD_LIBRARY_PATH
    $ ./zlib_test

#### Building Zlib with Dynamic Linking to Intel® IPP

Sometimes, you may want to build `libz` dynamic library linked dynamically to Intel® IPP libraries. This is, for example, if you know that `libz` will be used on a computer with pre-installed and properly set dynamic Intel® IPP libraries. Or, you may want to minimize file size of `libz` dynamic library, or, to decrease the total amount of RAM size required for several application(s) that use `libz`.

In this case, the following set of commands will build the `libz.so` library with dynamic linking to the Intel® IPP libraries (64-bit example).

    $ <Configure Intel(R) IPP environment>
    $ export CFLAGS="-m64 -DWITH_IPP -I$IPPROOT/include"
    $ export LDFLAGS="-L$IPPROOT/lib/intel64 -lippdc -lipps -lippcore"
    $ ./configure
    $ make shared

On macOS* system, the corresponding set of command will be:

    $ <Configure Intel(R) IPP environment>
    $ export CFLAGS="-m64 -DWITH_IPP -I$IPPROOT/include"
    $ export LDFLAGS="-L$IPPROOT/lib -lippdc -lipps -lippcore"
    $ ./configure
    $ make shared
    $ otool -L libz.dylib

On a target computer, before the `libz` library is used, you need to properly set paths to Intel® IPP dynamic libraries. On Linux* OS, do this by modifying the `LD_LIBRARY_PATH` environment variable. On macOS*, modify `DYLD_LIBRARY_PATH`. If you use `compilervars` script to set working environment, these environment variables are set automatically.

### Using Static `libz.a` Library to Build User Applications

During standard "configure/make" process the makefile creates the `libz.a` static library with `libz` compiled functions. You may want to use this static library with your application instead of using the `libz` shared object.

The static `libz.a` library contains unresolved external references to Intel® IPP library functions, so, when you build your application with the static `libz.a` library, you still need to solve the problem of linking to Intel® IPP libraries.

For static linking to Intel® IPP, use the following linking options:

    libz.a $IPPROOT/lib/<arch>/libippdc.a $IPPROOT/lib/<arch>/libipps.a $IPPROOT/lib/<arch>/libippcore.a

where <arch> is either `ia32` or `intel64` depending on kind of application (32-bit or 64-bit) you are going to build.

On macOS* the corresponding linking options are

    libz.a $IPPROOT/lib/libippdc.a $IPPROOT/lib/libipps.a $IPPROOT/lib/libippcore.a

For dynamic linking to Intel® IPP on Linux* OS, use the following options:

    libz.a -L$IPPROOT/lib/<arch> -lippdc -lipps -lippcore

On macOS*

    libz.a -L$IPPROOT/lib -lippdc -lipps -lippcore

## Downloading, Patching, and Build on Windows*

To download Zlib source code files for Windows*, you can use the same URL addresses as for Linux*. However, the patching and building processes on Windows* are different.

### Unpacking Zlib Tarball

To get the Zlib source code files on Windows* computer, you need to:

- Download the `.tar.gz` file from http://zlib.net

- Unpack the archive using any of available tar tools for Windows*. For example

  - Pre-built `tar` from Free Software Foundation (http://gnuwin32.sourceforge.net/packages/gtar.htm)

  - `7-zip` from http://www.7-zip.org/.

### Patching Original Zlib Files

There are no standard system tools for patching on Windows*, so you need to use 3-rd party tools. For example, http://gnuwin32.sourceforge.net/packages/patch.htm from Free Software Foundation.

With this tool you need to use the following command line options:

    > patch.exe -p1 --binary < <patch file location>

"--binary" command line option here is important to avoid confusion because of Linux*/Windows* "end-of-line" character difference.

Example for zlib-1.2.7.3 is below:

    > tar.exe xvf zlib-1.2.7.3.tar.gz
    > cd zlib-1.2.7.3
    > patch.exe -p1 --binary < ..\zlib-1.2.7.3.patch.bin
    patching file adler32.c
    patching file crc32.c
    patching file deflate.c
    patching file inflate.c
    patching file inftrees.h
    patching file trees.c

### Building Zlib With Intel® IPP on Windows*

To build Zlib on Windows*, use the "nmake" file `Makefile.msc` from the `zlib\win32` directory. To do that, start Microsoft* Visual Studio* command-line window that corresponds to a target application architecture (IA-32 or Intel® 64) and execute the following command from the root of Zlib source code tree (below are 32-bit examples).

Static linking:

    > nmake -f win32\Makefile.msc LOC="-DWITH_IPP -I\"%IPPROOT%\include\"" ^
           LDFLAGS="-nologo -incremental:no ^
           -opt:ref /LIBPATH:\"%IPPROOT%\lib\ia32\" ^
           ippdcmt.lib ippsmt.lib ippcoremt.lib"

Dynamic linking:

    > nmake -f win32\Makefile.msc LOC="-DWITH_IPP -I\"%IPPROOT%\include\"" ^
            LDFLAGS="-nologo -incremental:no ^
            -opt:ref /LIBPATH:\"%IPPROOT%\lib\ia32\" ^
            ippdc.lib ipps.lib ippcore.lib"

In the first case the "nmake" creates the `zlib1.dll` file not depending on dynamic Intel® IPP libraries. In the second, `zlib1.dll` depends on `ippdc.dll`, `ipps.dll` and `ippcore.dll` dynamic libraries.

### Building User Application with the Static `zlib.lib` library

When static linking is used, the Zlib build procedure, along with `zlib1.dll`, creates the static Zlib library `zlib.lib`. It can be used with application object files to be built statically.

If you want the Intel® IPP library to be linked to your application statically, you need to specify the following input files in the command line to Microsoft* linker:

    "%IPPROOT%\lib\<arch>\ippdcmt.lib"  "%IPPROOT%\lib\<arch>\ippsmt.lib"  "%IPPROOT%\lib\<arch>\ippcoremt.lib"

For dynamic linking, you need to choose other libraries from the Intel® IPP library directory:

    "%IPPROOT%\lib\<arch>\ippdc.lib"  "%IPPROOT%\lib\<arch>\ipps.lib"  "%IPPROOT%\lib\<arch>\ippcore.lib"

These libraries are just references to Intel® IPP dynamic libraries (DLLs).

Note | For Windows* build procedure it is important to use double quotes """ around `%IPPROOT%`, because by default Intel® IPP is installed to the "`Program Files(x86)`" directory. Its name contains spaces and, without quotes around, this name can confuse Microsoft* compiler and/or linker.

# How to Verify Zlib with Intel® Integrated Performance Primitives Library

For smoke testing or verification of the newly built library, you can use small tests/examples, which are supplied with the Zlib distribution.

## Verification on Linux* and macOS*

The makefile generated by the `configure` utility can be used to test library build with Makefile’s `test` target:

    $ make test

### Dynamic `libz` Library and Dynamic Tests

In general, command lines to build and test are (64-bit Linux* example on zlib 1.2.11):

    $ export CFLAGS="-m64 -DWITH_IPP -I$IPPROOT/include"
    $ export LDFLAGS="$IPPROOT/lib/intel64/libippdc.a \
            $IPPROOT/lib/intel64/libipps.a $IPPROOT/lib/intel64/libippcore.a"
    $ ./configure
    $ make testshared
    hello world
    zlib version 1.2.11 = 0x12b0, compile flags = 0xa9
    uncompress(): hello, hello!
    gzread(): hello, hello!
    gzgets() after gzseek:  hello!
    inflate(): hello, hello!
    large_inflate(): OK
    after inflateSync(): hello, hello!
    inflate with dictionary: hello, hello!
                    *** zlib shared test OK ***

### Static `libz` Library and Static Tests

For static tests you need to use slightly different approach, because in original Zlib makefile `LDFLAGS` variable is not used for static build. `TEST_LDFLAGS` argument in "make" command line must be used instead.

So, the command lines should be (32-bit Linux* example)

    $ export CFLAGS="-m32 -DWITH_IPP -I$IPPROOT/include"
    $ ./configure
    $ make teststatic TEST_LDFLAGS="libz.a $IPPROOT/lib/ia32/libippdc.a \
                         $IPPROOT/lib/ia32/libipps.a $IPPROOT/lib/ia32/libippcore.a"
    hello world
    zlib version 1.2.11 = 0x12b0, compile flags = 0x55
    uncompress(): hello, hello!
    gzread(): hello, hello!
    gzgets() after gzseek:  hello!
    inflate(): hello, hello!
    large_inflate(): OK
    after inflateSync(): hello, hello!
    inflate with dictionary: hello, hello!
                    *** zlib test OK ***

### Test `libz` on macOS*

Below are the same example command lines for macOS*

    $ export CFLAGS="-m64 -DWITH_IPP -I$IPPROOT/include"
    $ export LDFLAGS="$IPPROOT/lib/libippdc.a $IPPROOT/lib/libipps.a \
                      $IPPROOT/lib/libippcore.a"
    $ ./configure
    $ make testshared
    hello world
    zlib version 1.2.8 = 0x1280, compile flags = 0xa9
    uncompress(): hello, hello!
    gzread(): hello, hello!
    gzgets() after gzseek:  hello!
    inflate(): hello, hello!
    large_inflate(): OK
    after inflateSync(): hello, hello!
    inflate with dictionary: hello, hello!
                    *** zlib shared test OK ***

and

    $ export CFLAGS="-m32 -DWITH_IPP -I$IPPROOT/include"
    $ ./configure
    $ make teststatic TEST_LDFLAGS="libz.a $IPPROOT/lib/libippdc.a \
                         $IPPROOT/lib/libipps.a $IPPROOT/lib/libippcore.a"
    hello world
    zlib version 1.2.8 = 0x1280, compile flags = 0x95
    uncompress(): hello, hello!
    gzread(): hello, hello!
    gzgets() after gzseek:  hello!
    inflate(): hello, hello!
    large_inflate(): OK
    after inflateSync(): hello, hello!
    inflate with dictionary: hello, hello!
                    *** zlib test OK ***

## Verification on Windows*

Command line to build and test dynamic Zlib library on Windows* is

    nmake -f win32\Makefile.msc LOC="-DWITH_IPP -I\"%IPPROOT%\include\"" ^
        LDFLAGS="-nologo -incremental :no -opt:ref ^
        /LIBPATH:\"%IPPROOT%\lib\ia32\" ^
        ippdcmt.lib ippsmt.lib ippcoremt.lib" testdll

    Microsoft (R) Program Maintenance Utility Version 14.00.23506.0
    Copyright (C) Microsoft Corporation.  All rights reserved.

            example_d
    zlib version 1.2.6.1 = 0x1261, compile flags = 0x55
    uncompress(): hello, hello!
    gzread(): hello, hello!
    gzgets() after gzseek:  hello!
    inflate(): hello, hello!
    large_inflate(): OK
    after inflateSync(): hello, hello!
    inflate with dictionary: hello, hello!
            echo hello world | minigzip_d | minigzip_d -d
    hello world

# Using of Trained Deflate Tables in Fastest Compression Mode

In Intel® IPP version of Zlib, the fastest compression mode is performed using the simplified pattern of the search algorithm and special table for Huffman compression. By default, the static Huffman compression table is used. This table provides an average compression ratio suitable for all kinds of `deflate` input data.

If `deflate` is used to compress more or less uniform input data, for example texts, HTML files, or XML tables, you can significantly improve the compression ratio without losses to compression performance by using specific compression tables, which contain pre-built statistical data on such kind of information.

To generate and use trained deflate tables, the following functionality is added to the original Zlib distribution:

    #if defined(WITH_IPP) && defined(WITH_IPP_TABLES)
    ZEXTERN int ZEXPORT zzdeflateCreateStat OF((z_streamp));
    ZEXTERN int ZEXPORT zzdeflateGetTabLen OF((z_streamp));
    ZEXTERN int ZEXPORT zzdeflateGetTab OF((z_streamp, voidpf));
    ZEXTERN int ZEXPORT zzdeflateUseTab OF((z_streamp, voidpf, int));

    #define deflateCreateStat zzdeflateCreateStat
    #define deflateGetTabLen zzdeflateGetTabLen
    #define deflateGetTab zzdeflateGetTab
    #define deflateUseTab zzdeflateUseTab
    #endif

## New Intel® IPP Functions for Compression with Trained Deflate Tables

The new functionality is implemented in the scope of original `z_stream` objects. To use functions for statistics collection, table generation and table use, you need to have initialized `z_stream` objects.

Function names start with "zz" prefix and the functions are placed at the end of dynamic libraries (dynamic object .so on Linux* and DLL on Windows*). This prevents from breaking ordinal numbers of functions in dynamic library, though this usage model - ordinal numbers instead of function names - is not recommended by Zlib documentation. In `zlib.h` file the new function names are also redefined to make them more useful.

### `deflateCreateStat` Function

Syntax:

    int deflateCreateStat(z_streamp stream_ptr);

This function starts collecting of input data statistics for all subsequent calls to `deflate` function on the same stream. The `deflateCreateStat` function must be used with address of initialized stream as argument. The typical use is

        int      zstatus;
        z_stream stat_stream;

        stat_stream.zalloc = Z_NULL;
        stat_stream.zfree  = Z_NULL;
        zstatus            = deflateInit(&stat_stream, Z_IPP_FAST_COMPRESSION);
        zstatus            = deflateCreateStat(&stat_stream);

After initialization all calls to `deflate` function, besides performing compression itself, accumulate input data entropy statistics.

The return values are:

Value | Description
--- | ---
`Z_OK` | Success
`Z_STREAM_ERROR` | Argument is `NULL`, or stream is not initialized, or compression level is not `Z_IPP_FAST_COMPRESSION`
`Z_MEM_ERROR` | Function is not able to allocate internal arrays

### `deflateGetTabLen` Functon

Syntax:

    int deflateGetTabLen(z_streamp stream_ptr);

This function stops accumulating of input data statistics, calculates the size of trained table, generates the table and returns its length to calling application. The table is stored internally `z_stream` until `deflateGetTab` function is called.

The function accepts pointer to `z_stream` with accumulated statistics as its argument.

Return value is length of generated table. Application must allocate dynamic memory buffer of that length (usually, length is about 1K bytes) and, then must call `deflateGetTab` function to copy generated table from `z_stream` to external buffer for subsequent use.

Return values are:

Value | Description
--- | ---
> 0 | Size of generated table
`Z_STREAM_ERROR` | Argument is `NULL`, or stream is not initialized, or stream with no statistics specified
0 | Internal Intel® IPP function failed

### `deflateGetTab` Function

Syntax:

    int deflateGetTab(z_streamp stream_ptr, voidpf table_ptr);

This function copies generated deflate table from `z_stream` internal buffer to buffer allocated in application memory for external use. The arguments are: `s_streamp` pointer to `z_stream` with generated table, `voidpf` is a `void*` pointer to external buffer.

The typical use of `deflateGetTabLen` and `deflateGetTab` function is:

        int     tab_len;
        void*   tab_ptr;
        int     zstatus;
        /* Accimulate statistics for part of input data... */
        tab_len = deflateGetTabLen(&statistic_stream);
        tab_ptr = malloc(tab_len);
        zstatus = deflateGetTab(&statistic_stream, tab_ptr);

The return values are:

Value | Description
--- | ---
`Z_OK` | Success
`Z_STREAM_ERROR` | Argument is `NULL`, or stream is not initialized, or stream with no statistics specified, or table is not generated for stream

### `deflateUseTab` Function

Syntax:

    int deflateUseTab(z_streamp stream_ptr, voidpf table_ptr, int table_len);

This function copies buffer with trained table to `z_stream` internal memory. During the next `deflate` function calls for that stream, this trained table will be used to increase ratio of the compression operations.

Return values are:

Value | Description
--- | ---
`Z_OK` | Success
`Z_STREAM_ERROR` | Argument is `NULL`, or stream is not initialized, or table pointer is `NULL`, or table length is zero

The operations with `z_stream`, associated with trained table, are as usual.

Note | Currently, `z_stream` object is not designed to change trained deflate tables "on-the-run", so to apply another trained table to compress other kind of input data, you need to close stream with previous table and open a new stream for another trained table.

## Compression Ratio Results of Intel® IPP Zlib with Trained Deflate Tables

The table below shows the compression ratio and performance of deflating of files from Calgary* data compression corpus without- and with trained compression table(s).

- CPU: Intel® Core™ i5-4300U @ 2.5 GHz on Windows* 8.1 Enterprise, 64 bits

- Compression ratio is calculated as <file size before compression> / <file size after compression>

- Compression performance is <file size before compression> / <compression time> in megabytes per second

- Compression level is -2 (`Z_IPP_FAST_COMPRESSION`).

File | Default Ratio | Default Compression Performance | Ratio with Table | Performance with Table
--- | --- | --- | --- | ---
bib | 2.15 | 176 | 2.60 | 179
book1 | 1.71 | 126 | 2.19 | 158
book2 | 2.06 | 177 | 2.54 | 186
geo | 1.10 | 93 | 1.46 | 96
news | 1.97 | 130 | 2.37 | 148
obj1 | 1.77 | 206 | 1.97 | 222
obj2 | 2.31 | 178 | 2.59 | 177
paper1 | 2.06 | 179 | 2.50 | 186
paper2 | 1.93 | 157 | 2.42 | 172
paper3 | 1.84 | 159 | 2.30 | 186
paper4 | 1.77 | 177 | 2.23 | 240
paper5 | 1.81 | 176 | 2.23 | 246
paper6 | 2.08 | 200 | 2.52 | 214
pic | 6.76 | 534 | 8.28 | 519
progc | 2.19 | 203 | 2.62 | 211
progl | 3.03 | 245 | 3.63 | 252
progp | 3.13 | 273 | 3.68 | 288
trans | 3.45 | 307 | 3.98 | 309

The data above shows the best result when a separate table is generated for each Calgary* file and this table is used for compression. If you generate a trained table for a set of files, the resulting compression ratio will be less by some points. The compression performance will likely be the same.

## How to Create and Use Trained Deflate Tables

Before you start using a trained deflate table, you need to build it. The build process includes two phases:

1) Accumulation of statistics on input data

2) Generation of the trained table.

Accumulation of statistics can be done in parallel with ordinary compression.

For example,

    #define WITH_IPP
    #define WITH_IPP_TABLES
    #include "zlib.h"
    ...
        z_stream def_str;
        z_stream table_str;
        int      tab_len;
        void*    p_table;

        def_str.zalloc = Z_NULL;
        def_str.zfree = Z_NULL;
        deflateInit(&def_str, Z_IPP_FAST_COMPRESSION);
        deflateCreateStat(&def_str);
        def_str.next_out = buffer_out;
        def_str.avail_out = sizeof(buffer_out);

        def_str.next_in = buffer1;
        def_str.avail_in = sizeof(buffer1);
        deflate(&def_str, Z_NO_FLUSH);

        def_str.next_in = buffer2;
        def_str.avail_in = sizeof(buffer2);
        deflate(&def_str, Z_FULL_FLUSH);
        /* Using of buffer_out is possible,
           it contains data compressed with fastest level,
           though without trained table */

        /* Generate and save trained table */
        tab_len = deflateGetTabLen(&def_str);
        p_table = malloc(tab_len);
        deflateGetTab(&def_str, p_table);

        /* Close statistics stream */
        deflateEnd(&def_str);

        /* Open a new stream for compression with trained table */
        table_str.zalloc = Z_NULL;
        table_str.zfree = Z_NULL;
        deflateInit(&table_str, Z_IPP_FAST_COMPRESSION);
        deflateUseTab(&table_str, p_table, tab_len);

        /* Then use table_str for compression as usually */

The definition of `WITH_IPP` and `WITH_IPP_TABLES` is important in the example above, because patched `zlib.h` file contains some extensions in conditional compilation constructions.

You can train deflate tables for different kind of data - text, binary, HTMLs - and re-use them on appropriate data.

Note | Important Trained deflate table is an experimental mode in Intel® IPP Zlib, so use this feature with some caution.

Note | Don’t pass much data through statistics accumulation There is no automatic checking of data statistics, so if too much data is compressed in the "statistics accumulation" mode, the statistic array overflow may happen. In most cases about 1 MB of input data is enough for good trained table generation.

    ...
        deflateInit(&def_str, Z_IPP_FAST_COMPRESSION);
        deflateCreateStat(&def_str);
    ...
        while(def_str.total_in < (1024 * 1024))
        {
            /* deflate... */
        }
        /* Generate and save trained table */
        tab_len = deflateGetTabLen(&def_str);
        p_table = malloc(tab_len);
        deflateGetTab(&def_str, p_table);

Note | Don’t use trained deflate table for inappropriate input data Data compression is a data-driven process, so, in some cases, mixing of trained table for some kind of data with compression of different kind of data may decrease the compression ratio, in some cases the ratio could be even less than 1.

        deflateUseTab(&table_str, text_table_ptr, text_table_len);
        ...
        table_str.next_in = buffer_for_binary_data;
        ...
        deflate(&table_str, ...)    /* Don't do this! */

For example, below is table with compression ratio of Calgary* files, compressed with table trained on all Calgary* files (Calgary*-generated table).

File | Default Ratio | Ratio with File-generated Table | Ratio with Calgary*-generated table
--- | --- | --- | ---
bib | 2.15 | 2.60 | 2.52
book1 | 1.71 | 2.19 | 2.13
book2 | 2.06 | 2.54 | 2.48
geo | 1.10 | 1.46 | 1.16
news | 1.97 | 2.37 | 2.31
obj1 | 1.77 | 1.97 | 1.67
obj2 | 2.31 | 2.59 | 2.43
paper1 | 2.06 | 2.50 | 2.43
paper2 | 1.93 | 2.42 | 2.36
paper3 | 1.84 | 2.30 | 2.24
paper4 | 1.77 | 2.23 | 2.10
paper5 | 1.81 | 2.23 | 2.10
paper6 | 2.08 | 2.52 | 2.44
pic | 6.76 | 8.28 | 6.75
progc | 2.19 | 2.62 | 2.52
progl | 3.03 | 3.63 | 3.50
progp | 3.13 | 3.68 | 3.49
trans | 3.45 | 3.98 | 3.81

You see that, for example, for the `pic` file the compression ratio with Calgary* generated table is worse than if no tables are used at all.

## How to Build Zlib with Intel® Integrated Performance Primitives Library for Trained Deflate Tables

In order to build Zlib library with Intel® IPP functionality for trained deflate table you may use the procedure described in corresponding How to Build Zlib with Intel® Integrated Performance Primitives Library chapter. The only difference is, that you need to specify additional definition `-DWITH_IPP_TABLES` together with definition `-DWITH_IPP`.

So, correct compiler options for Linux* must be

    $ export CFLAGS="-DWITH_IPP -DWITH_IPP_TABLES -I$IPPROOT/include"

On Windows* you need only to add a new definition of `WITH_IPP_TABLES`, but also add four new functions as exported symbols of dynamic library.

Static Intel® IPP linking:

    > nmake -f win32\Makefile.msc LOC="-DWITH_IPP -DWITH_IPP_TABLES -I\"%IPPROOT%\include\"" ^
           LDFLAGS="-nologo -incremental:no -opt:ref ^
           /EXPORT:zzdeflateCreateStat /EXPORT:zzdeflateGetTabLen /EXPORT:zzdeflateGetTab ^
           /EXPORT:zzdeflateUseTab ^
           /LIBPATH:\"%IPPROOT%\lib\ia32\" ^
           ippdcmt.lib ippsmt.lib ippcoremt.lib"

Dynamic Intel® IPP linking:

    > nmake -f win32\Makefile.msc LOC="-DWITH_IPP -DWITH_IPP_TABLES -I\"%IPPROOT%\include\"" ^
           LDFLAGS="-nologo -incremental:no -opt:ref ^
           /EXPORT:zzdeflateCreateStat /EXPORT:zzdeflateGetTabLen /EXPORT:zzdeflateGetTab ^
           /EXPORT:zzdeflateUseTab ^
           /LIBPATH:\"%IPPROOT%\lib\ia32\" ^
           ippdc.lib ipps.lib ippcore.lib"

# Technical Support

If you did not register your Intel® software product during installation, please do so now at the Intel® Software Development Products Registration Center. Registration entitles you to free technical support, product updates and upgrades for the duration of the support term.

For general information about Intel technical support, product updates, user forums, FAQs, tips and tricks, and other support questions, please visit (http://www.intel.com/software/products/support).

Note | If your distributor provides technical support for this product, please contact them rather than Intel.

For technical information about the Intel® IPP library, including FAQ’s, tips and tricks, and other support information, please visit the Intel® IPP forum: (http://software.intel.com/en-us/forums/intel-integrated-performance-primitives) and browse the Intel® IPP support page: https://software.intel.com/en-us/intel-ipp-support/.