Disclaimer

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.
A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.

Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined." Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.
The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.
Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.
Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm

Intel processor numbers are not a measure of performance. Processor numbers differentiate features within each processor family, not across different processor families. Go to: http://www.intel.com/products/processor_number/

BlueMoon, BunnyPeople, Celeron, Celeron Inside, Centrino, Centrino Inside, Core Inside, i960, Intel, the Intel logo, Intel Atom, Intel Atom Inside, Intel Core, Intel Inside, Intel Inside logo, Intel NetBurst, Intel NetMerge, Intel NetStructure, Intel SingleDriver, Intel SpeedStep, Intel Sponsors of Tomorrow., the Intel Sponsors of Tomorrow. logo, Intel StrataFlash, Intel Viiv, Intel vPro, Intel XScale, InTru, the InTru logo, InTru soundmark, Itanium, Itanium Inside, MCS, MMX, Moblin, Pentium, Pentium Inside, skoool, the skoool logo, Sound Mark, The Journey Inside, vPro Inside, VTune, Xeon, and Xeon Inside are trademarks of Intel Corporation in the U.S. and other countries.
* Other names and brands may be claimed as the property of others.

Microsoft, Windows, Visual Studio, Visual C++, and the Windows logo are trademarks, or registered trademarks of Microsoft Corporation in the United States and/or other countries.

Java is a registered trademark of Oracle and/or its affiliates.

©Intel Corporation. All rights reserved.

Optimization Notice

Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel.

Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.

Notice revision #20110804

License Definitions

By downloading and installing this product, you hereby agree that the accompanying materials are being provided to you under the terms and conditions of the End User License Agreement for the Intel® Integrated Performance Primitives (Intel® IPP) product previously accepted by you.

System Requirements

Hardware requirements:

Software requirements:

For more information please see https://software.intel.com/en-us/ipp [Intel® Integrated Performance Primitives].

What’s New

The following updates have been made to Intel® IPP functions for Zlib:

    deflateInit(&deflate_stream, -2);

You can use symbolic name Z_IPP_FAST_COMPRESSION, which is equal to -2:

    deflateInit(&deflate_stream, Z_IPP_FAST_COMPRESSION);
ZEXTERN int ZEXPORT deflateTune OF((z_streamp strm,
                                    int good_length,
                                    int max_lazy,
                                    int nice_length,
                                    int max_chain));
Fine tune deflate's internal compression parameters. This should only be used by
someone who understands the algorithm used by Zlib's +deflate+ for searching for the
best matching string, and even then only by the most fanatic optimizer trying
to squeeze out the last compressed bit for their specific input data.

Read the deflate.c source code for the meaning of the max_lazy, good_length,
nice_length, and max_chain parameters.

The described function arguments now affect Intel® IPP data compression functions.

You can find the default sets of string matching parameters in the configuration_table array inside deflate.c source code file.

How to Build Zlib with Intel® Integrated Performance Primitives Library

The data compression domain of the Intel® IPP library contains several functions that can speed up the well-known Zlib library (http://zlib.net) in both data compression and decompression operations. For the detailed description of these functions, refer to the Volume 1: "Signal Processing/Data Compression Functions/Dictionary-Based Compression Functions/ZLIB Coding Functions" section of the Intel® IPP Developer Reference.

To use these functions, before building the library you need to replace the original source code of several Zlib functions with Intel® IPP functions calls.

This document provides instructions on how to prepare the Zlib source code and build the library.

Prerequisites

The Zlib repository contains several stable versions of the library: v.1.2.6.1, v.1.2.7.3, v.1.2.8 and v.1.2.11. You may want to update and speed up a particular Zlib version, which is used in your application, or on your operating system. For that, Intel® IPP product provides four source code patch files applicable to specific versions of Zlib.

Downloading, Patching and Build on Linux* and macOS*

Downloading Zlib Source Code

Before building the library, you need to download Zlib source code files from the Zlib site (http://zlib.net). The following table provides links to specific Zlib versions that can be updated with Intel® IPP library function calls.

Download the tar file from the specified address to a working directory. Unpack the downloaded archive with the following command on Linux*/macOS* systems.

$ tar xvf <archive file name>

This command will create the zlib-<version> folder in your working directory, where <version> is selected Zlib version number from 1.2.6.1 to 1.2.11.

Patching the Source Code

  1. Change directory to the zlib-<version> folder.

  2. Apply the corresponding source code patch file zlib-<version>.patch.bin from Intel® IPP product distribution with the following command

$ patch -p1 < "path to corresponding patch file"

For example, to patch Zlib v.1.2.6.1, go to zlib-1.2.6.1 directory and run the following command

$ patch -p1 < ../zlib-1.2.6.1.patch.bin

If the patching process completed successfully, you will see the following messages:

patching file adler32.c
patching file crc32.c
patching file deflate.c
patching file inflate.c
patching file inftrees.h
patching file trees.c

If something goes wrong, for example, if the adler32.c file has been changed (nonoriginal), you will see the diagnostics messages:

patching file adler32.c
Hunk #1 FAILED at 5.
patching file crc32.c
patching file deflate.c
patching file inflate.c
patching file inftrees.h
patching file trees.c

Configuring and Building Zlib Library

After Zlib source code files are patched successfully, you need to create a makefile for building process. Basically, you need to

  • Add the WITH_IPP definition to compiler command line

  • Add the $IPPROOT/include directory to the list of header files search for compiler

  • Add Intel® IPP libraries to the list of linker input files.

On Linux* OS, you can implement the above steps with the standard Zlib configure/make sequence and the following commands (32-bit example):

# Build dynamic Zlib library with static Intel(R) IPP linkage on Linux*
$ export CFLAGS="-m32 -DWITH_IPP -I$IPPROOT/include"
$ export LDFLAGS="$IPPROOT/lib/ia32/libippdc.a $IPPROOT/lib/ia32/libipps.a $IPPROOT/lib/ia32/libippcore.a"
$ ./configure
$ make shared

Here, the -m32 option specifies IA-32 target CPU architecture. For Intel® 64 architecture the compiler option should be -m64.

The above commands assume that IPPROOT environment variable points to head of Intel® IPP location.

Configure your Intel® IPP working environment according to "Get Started with Intel® Integrated Performance Primitives" document.

Note
The conditional compilation statements, used for Zlib source code files modification, allow to build original version of Zlib from the same source code files. For that, you need to build Zlib library without modification of CFLAGS and LDFLAGS. If WITH_IPP definition doesn’t exist in compiler options, Zlib source code files are compiled as unpatched.

Static and Dynamic Linking to Intel® IPP Libraries

When building the libz dynamic shared object, you can use either static or dynamic linking to Intel® IPP libraries. Different linking modes have their advantages and disadvantages. For example, static linking to Intel® IPP provides target object (libz.so, or libz.dylib in our case) independence from environment, easy transfer from one computer to another. On the other hand, statically linked object has bigger size and can’t benefit from newer (updated) version of Intel® IPP without re-linking.

The libz dynamic libraries, linked statically with Intel® IPP, are self-sufficient. It means that when you build an application using these libraries, you don’t need to add Intel® IPP libraries to the linker command line (unless your application uses Intel® IPP apart from libz). All references to Intel® IPP functions are already resolved, and required Intel® IPP functions code is inside the libz dynamic library.

Build Zlib with Static Linking to Intel® IPP

On Linux* OS, when the below linker flags are specified by the LDFLAGS environment variable, Zlib is linked to Intel® IPP libraries (.a files) statically:

$ export LDFLAGS="$IPPROOT/lib/intel64/libippdc.a $IPPROOT/lib/intel64/libipps.a \
                  $IPPROOT/lib/intel64/libippcore.a"

You can check the independence of the resulting dynamic library libz.so from Intel® IPP libraries with the "ldd" tool.

For example,

$ ldd libz.so
        linux-vdso.so.1 =>  (0x00007fff885fe000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fb169e38000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb16a503000)

You can see that the libz.so dynamic library depends on system dynamic libraries only. The size of the resulting libz library in this case will be more than 700 KBytes because it contains optimizations for all CPUs supported by Intel® IPP.

On macOS*, the similar tool is called "otool". For example,

$ otool -L libz.dylib
libz.dylib:
        /usr/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

You can use the following command to build an application with such kind of library:

$ cc -o zlib_test -Izlib/include zlib_test.c -L. -lz

Here, zlib/include is a directory where zlib header files reside (in most cases zlib.h and zconf.h are required), "-L." means that Linux* linker will be searching next libraries in the current directory, "-lz" means that the linker will look for the libz.so file (shared linking is default method on most Linux* systems).

To run built application on Linux* system, you need to modify the LD_LIBRARY_PATH environment variable, which describes the search path for dynamic libraries:

$ export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH
$ ./zlib_test

On macOS* systems the corresponding environment variable is DYLD_LIBRARY_PATH:

$ export DYLD_LIBRARY_PATH=.:$DYLD_LIBRARY_PATH
$ ./zlib_test

Building Zlib with Dynamic Linking to Intel® IPP

Sometimes, you may want to build libz dynamic library linked dynamically to Intel® IPP libraries. This is, for example, if you know that libz will be used on a computer with pre-installed and properly set dynamic Intel® IPP libraries. Or, you may want to minimize file size of libz dynamic library, or, to decrease the total amount of RAM size required for several application(s) that use libz.

In this case, the following set of commands will build the libz.so library with dynamic linking to the Intel® IPP libraries (64-bit example).

$ <Configure Intel(R) IPP environment>
$ export CFLAGS="-m64 -DWITH_IPP -I$IPPROOT/include"
$ export LDFLAGS="-L$IPPROOT/lib/intel64 -lippdc -lipps -lippcore"
$ ./configure
$ make shared

On macOS* system, the corresponding set of command will be:

$ <Configure Intel(R) IPP environment>
$ export CFLAGS="-m64 -DWITH_IPP -I$IPPROOT/include"
$ export LDFLAGS="-L$IPPROOT/lib -lippdc -lipps -lippcore"
$ ./configure
$ make shared
$ otool -L libz.dylib

On a target computer, before the libz library is used, you need to properly set paths to Intel® IPP dynamic libraries. On Linux* OS, do this by modifying the LD_LIBRARY_PATH environment variable. On macOS*, modify DYLD_LIBRARY_PATH. If you use compilervars script to set working environment, these environment variables are set automatically.

Using Static libz.a Library to Build User Applications

During standard "configure/make" process the makefile creates the libz.a static library with libz compiled functions. You may want to use this static library with your application instead of using the libz shared object.

The static libz.a library contains unresolved external references to Intel® IPP library functions, so, when you build your application with the static libz.a library, you still need to solve the problem of linking to Intel® IPP libraries.

For static linking to Intel® IPP, use the following linking options:

libz.a $IPPROOT/lib/<arch>/libippdc.a $IPPROOT/lib/<arch>/libipps.a $IPPROOT/lib/<arch>/libippcore.a

where <arch> is either ia32 or intel64 depending on kind of application (32-bit or 64-bit) you are going to build.

On macOS* the corresponding linking options are

libz.a $IPPROOT/lib/libippdc.a $IPPROOT/lib/libipps.a $IPPROOT/lib/libippcore.a

For dynamic linking to Intel® IPP on Linux* OS, use the following options:

libz.a -L$IPPROOT/lib/<arch> -lippdc -lipps -lippcore

On macOS*

libz.a -L$IPPROOT/lib -lippdc -lipps -lippcore

Downloading, Patching, and Build on Windows*

To download Zlib source code files for Windows*, you can use the same URL addresses as for Linux*. However, the patching and building processes on Windows* are different.

Unpacking Zlib Tarball

To get the Zlib source code files on Windows* computer, you need to:

Patching Original Zlib Files

There are no standard system tools for patching on Windows*, so you need to use 3-rd party tools. For example, http://gnuwin32.sourceforge.net/packages/patch.htm from Free Software Foundation.

With this tool you need to use the following command line options:

> patch.exe -p1 --binary < <patch file location>

"--binary" command line option here is important to avoid confusion because of Linux*/Windows* "end-of-line" character difference.

Example for zlib-1.2.7.3 is below:

> tar.exe xvf zlib-1.2.7.3.tar.gz
> cd zlib-1.2.7.3
> patch.exe -p1 --binary < ..\zlib-1.2.7.3.patch.bin
patching file adler32.c
patching file crc32.c
patching file deflate.c
patching file inflate.c
patching file inftrees.h
patching file trees.c

Building Zlib With Intel® IPP on Windows*

To build Zlib on Windows*, use the "nmake" file Makefile.msc from the zlib\win32 directory. To do that, start Microsoft* Visual Studio* command-line window that corresponds to a target application architecture (IA-32 or Intel® 64) and execute the following command from the root of Zlib source code tree (below are 32-bit examples).

Static linking:

> nmake -f win32\Makefile.msc LOC="-DWITH_IPP -I\"%IPPROOT%\include\"" ^
       LDFLAGS="-nologo -incremental:no ^
       -opt:ref /LIBPATH:\"%IPPROOT%\lib\ia32\" ^
       ippdcmt.lib ippsmt.lib ippcoremt.lib"

Dynamic linking:

> nmake -f win32\Makefile.msc LOC="-DWITH_IPP -I\"%IPPROOT%\include\"" ^
        LDFLAGS="-nologo -incremental:no ^
        -opt:ref /LIBPATH:\"%IPPROOT%\lib\ia32\" ^
        ippdc.lib ipps.lib ippcore.lib"

In the first case the "nmake" creates the zlib1.dll file not depending on dynamic Intel® IPP libraries. In the second, zlib1.dll depends on ippdc.dll, ipps.dll and ippcore.dll dynamic libraries.

Building User Application with the Static zlib.lib library

When static linking is used, the Zlib build procedure, along with zlib1.dll, creates the static Zlib library zlib.lib. It can be used with application object files to be built statically.

If you want the Intel® IPP library to be linked to your application statically, you need to specify the following input files in the command line to Microsoft* linker:

"%IPPROOT%\lib\<arch>\ippdcmt.lib"  "%IPPROOT%\lib\<arch>\ippsmt.lib"  "%IPPROOT%\lib\<arch>\ippcoremt.lib"

For dynamic linking, you need to choose other libraries from the Intel® IPP library directory:

"%IPPROOT%\lib\<arch>\ippdc.lib"  "%IPPROOT%\lib\<arch>\ipps.lib"  "%IPPROOT%\lib\<arch>\ippcore.lib"

These libraries are just references to Intel® IPP dynamic libraries (DLLs).

Note
For Windows* build procedure it is important to use double quotes """ around %IPPROOT%, because by default Intel® IPP is installed to the "Program Files(x86)" directory. Its name contains spaces and, without quotes around, this name can confuse Microsoft* compiler and/or linker.

How to Verify Zlib with Intel® Integrated Performance Primitives Library

For smoke testing or verification of the newly built library, you can use small tests/examples, which are supplied with the Zlib distribution.

Verification on Linux* and macOS*

The makefile generated by the configure utility can be used to test library build with Makefile’s test target:

$ make test

Dynamic libz Library and Dynamic Tests

In general, command lines to build and test are (64-bit Linux* example on zlib 1.2.11):

$ export CFLAGS="-m64 -DWITH_IPP -I$IPPROOT/include"
$ export LDFLAGS="$IPPROOT/lib/intel64/libippdc.a \
        $IPPROOT/lib/intel64/libipps.a $IPPROOT/lib/intel64/libippcore.a"
$ ./configure
$ make testshared
hello world
zlib version 1.2.11 = 0x12b0, compile flags = 0xa9
uncompress(): hello, hello!
gzread(): hello, hello!
gzgets() after gzseek:  hello!
inflate(): hello, hello!
large_inflate(): OK
after inflateSync(): hello, hello!
inflate with dictionary: hello, hello!
                *** zlib shared test OK ***

Static libz Library and Static Tests

For static tests you need to use slightly different approach, because in original Zlib makefile LDFLAGS variable is not used for static build. TEST_LDFLAGS argument in "make" command line must be used instead.

So, the command lines should be (32-bit Linux* example)

$ export CFLAGS="-m32 -DWITH_IPP -I$IPPROOT/include"
$ ./configure
$ make teststatic TEST_LDFLAGS="libz.a $IPPROOT/lib/ia32/libippdc.a \
                     $IPPROOT/lib/ia32/libipps.a $IPPROOT/lib/ia32/libippcore.a"
hello world
zlib version 1.2.11 = 0x12b0, compile flags = 0x55
uncompress(): hello, hello!
gzread(): hello, hello!
gzgets() after gzseek:  hello!
inflate(): hello, hello!
large_inflate(): OK
after inflateSync(): hello, hello!
inflate with dictionary: hello, hello!
                *** zlib test OK ***

Test libz on macOS*

Below are the same example command lines for macOS*

$ export CFLAGS="-m64 -DWITH_IPP -I$IPPROOT/include"
$ export LDFLAGS="$IPPROOT/lib/libippdc.a $IPPROOT/lib/libipps.a \
                  $IPPROOT/lib/libippcore.a"
$ ./configure
$ make testshared
hello world
zlib version 1.2.8 = 0x1280, compile flags = 0xa9
uncompress(): hello, hello!
gzread(): hello, hello!
gzgets() after gzseek:  hello!
inflate(): hello, hello!
large_inflate(): OK
after inflateSync(): hello, hello!
inflate with dictionary: hello, hello!
                *** zlib shared test OK ***

and

$ export CFLAGS="-m32 -DWITH_IPP -I$IPPROOT/include"
$ ./configure
$ make teststatic TEST_LDFLAGS="libz.a $IPPROOT/lib/libippdc.a \
                     $IPPROOT/lib/libipps.a $IPPROOT/lib/libippcore.a"
hello world
zlib version 1.2.8 = 0x1280, compile flags = 0x95
uncompress(): hello, hello!
gzread(): hello, hello!
gzgets() after gzseek:  hello!
inflate(): hello, hello!
large_inflate(): OK
after inflateSync(): hello, hello!
inflate with dictionary: hello, hello!
                *** zlib test OK ***

Verification on Windows*

Command line to build and test dynamic Zlib library on Windows* is

nmake -f win32\Makefile.msc LOC="-DWITH_IPP -I\"%IPPROOT%\include\"" ^
    LDFLAGS="-nologo -incremental :no -opt:ref ^
    /LIBPATH:\"%IPPROOT%\lib\ia32\" ^
    ippdcmt.lib ippsmt.lib ippcoremt.lib" testdll

Microsoft (R) Program Maintenance Utility Version 14.00.23506.0
Copyright (C) Microsoft Corporation.  All rights reserved.

        example_d
zlib version 1.2.6.1 = 0x1261, compile flags = 0x55
uncompress(): hello, hello!
gzread(): hello, hello!
gzgets() after gzseek:  hello!
inflate(): hello, hello!
large_inflate(): OK
after inflateSync(): hello, hello!
inflate with dictionary: hello, hello!
        echo hello world | minigzip_d | minigzip_d -d
hello world

Using of Trained Deflate Tables in Fastest Compression Mode

In Intel® IPP version of Zlib, the fastest compression mode is performed using the simplified pattern of the search algorithm and special table for Huffman compression. By default, the static Huffman compression table is used. This table provides an average compression ratio suitable for all kinds of deflate input data.

If deflate is used to compress more or less uniform input data, for example texts, HTML files, or XML tables, you can significantly improve the compression ratio without losses to compression performance by using specific compression tables, which contain pre-built statistical data on such kind of information.

To generate and use trained deflate tables, the following functionality is added to the original Zlib distribution:

#if defined(WITH_IPP) && defined(WITH_IPP_TABLES)
ZEXTERN int ZEXPORT zzdeflateCreateStat OF((z_streamp));
ZEXTERN int ZEXPORT zzdeflateGetTabLen OF((z_streamp));
ZEXTERN int ZEXPORT zzdeflateGetTab OF((z_streamp, voidpf));
ZEXTERN int ZEXPORT zzdeflateUseTab OF((z_streamp, voidpf, int));

#define deflateCreateStat zzdeflateCreateStat
#define deflateGetTabLen zzdeflateGetTabLen
#define deflateGetTab zzdeflateGetTab
#define deflateUseTab zzdeflateUseTab
#endif

New Intel® IPP Functions for Compression with Trained Deflate Tables

The new functionality is implemented in the scope of original z_stream objects. To use functions for statistics collection, table generation and table use, you need to have initialized z_stream objects.

Function names start with "zz" prefix and the functions are placed at the end of dynamic libraries (dynamic object .so on Linux* and DLL on Windows*). This prevents from breaking ordinal numbers of functions in dynamic library, though this usage model - ordinal numbers instead of function names - is not recommended by Zlib documentation. In zlib.h file the new function names are also redefined to make them more useful.

deflateCreateStat Function

Syntax:

int deflateCreateStat(z_streamp stream_ptr);

This function starts collecting of input data statistics for all subsequent calls to deflate function on the same stream. The deflateCreateStat function must be used with address of initialized stream as argument. The typical use is

    int      zstatus;
    z_stream stat_stream;

    stat_stream.zalloc = Z_NULL;
    stat_stream.zfree  = Z_NULL;
    zstatus            = deflateInit(&stat_stream, Z_IPP_FAST_COMPRESSION);
    zstatus            = deflateCreateStat(&stat_stream);

After initialization all calls to deflate function, besides performing compression itself, accumulate input data entropy statistics.

The return values are:

Value Description

Z_OK

Success

Z_STREAM_ERROR

Argument is NULL, or stream is not initialized, or compression level is not Z_IPP_FAST_COMPRESSION

Z_MEM_ERROR

Function is not able to allocate internal arrays

deflateGetTabLen Functon

Syntax:

int deflateGetTabLen(z_streamp stream_ptr);

This function stops accumulating of input data statistics, calculates the size of trained table, generates the table and returns its length to calling application. The table is stored internally z_stream until deflateGetTab function is called.

The function accepts pointer to z_stream with accumulated statistics as its argument.

Return value is length of generated table. Application must allocate dynamic memory buffer of that length (usually, length is about 1K bytes) and, then must call deflateGetTab function to copy generated table from z_stream to external buffer for subsequent use.

Return values are:

Value Description

> 0

Size of generated table

Z_STREAM_ERROR

Argument is NULL, or stream is not initialized, or stream with no statistics specified

0

Internal Intel® IPP function failed

deflateGetTab Function

Syntax:

int deflateGetTab(z_streamp stream_ptr, voidpf table_ptr);

This function copies generated deflate table from z_stream internal buffer to buffer allocated in application memory for external use. The arguments are: s_streamp pointer to z_stream with generated table, voidpf is a void* pointer to external buffer.

The typical use of deflateGetTabLen and deflateGetTab function is:

    int     tab_len;
    void*   tab_ptr;
    int     zstatus;
    /* Accimulate statistics for part of input data... */
    tab_len = deflateGetTabLen(&statistic_stream);
    tab_ptr = malloc(tab_len);
    zstatus = deflateGetTab(&statistic_stream, tab_ptr);

The return values are:

Value Description

Z_OK

Success

Z_STREAM_ERROR

Argument is NULL, or stream is not initialized, or stream with no statistics specified, or table is not generated for stream

deflateUseTab Function

Syntax:

int deflateUseTab(z_streamp stream_ptr, voidpf table_ptr, int table_len);

This function copies buffer with trained table to z_stream internal memory. During the next deflate function calls for that stream, this trained table will be used to increase ratio of the compression operations.

Return values are:

Value Description

Z_OK

Success

Z_STREAM_ERROR

Argument is NULL, or stream is not initialized, or table pointer is NULL, or table length is zero

The operations with z_stream, associated with trained table, are as usual.

Note
Currently, z_stream object is not designed to change trained deflate tables "on-the-run", so to apply another trained table to compress other kind of input data, you need to close stream with previous table and open a new stream for another trained table.

Compression Ratio Results of Intel® IPP Zlib with Trained Deflate Tables

The table below shows the compression ratio and performance of deflating of files from Calgary* data compression corpus without- and with trained compression table(s).

  • CPU: Intel® Core™ i5-4300U @ 2.5 GHz on Windows* 8.1 Enterprise, 64 bits

  • Compression ratio is calculated as <file size before compression> / <file size after compression>

  • Compression performance is <file size before compression> / <compression time> in megabytes per second

  • Compression level is -2 (Z_IPP_FAST_COMPRESSION).

File Default Ratio Default Compression Performance Ratio with Table Performance with Table

bib

2.15

176

2.60

179

book1

1.71

126

2.19

158

book2

2.06

177

2.54

186

geo

1.10

93

1.46

96

news

1.97

130

2.37

148

obj1

1.77

206

1.97

222

obj2

2.31

178

2.59

177

paper1

2.06

179

2.50

186

paper2

1.93

157

2.42

172

paper3

1.84

159

2.30

186

paper4

1.77

177

2.23

240

paper5

1.81

176

2.23

246

paper6

2.08

200

2.52

214

pic

6.76

534

8.28

519

progc

2.19

203

2.62

211

progl

3.03

245

3.63

252

progp

3.13

273

3.68

288

trans

3.45

307

3.98

309

The data above shows the best result when a separate table is generated for each Calgary* file and this table is used for compression. If you generate a trained table for a set of files, the resulting compression ratio will be less by some points. The compression performance will likely be the same.

How to Create and Use Trained Deflate Tables

Before you start using a trained deflate table, you need to build it. The build process includes two phases:

1) Accumulation of statistics on input data

2) Generation of the trained table.

Accumulation of statistics can be done in parallel with ordinary compression.

For example,

#define WITH_IPP
#define WITH_IPP_TABLES
#include "zlib.h"
...
    z_stream def_str;
    z_stream table_str;
    int      tab_len;
    void*    p_table;

    def_str.zalloc = Z_NULL;
    def_str.zfree = Z_NULL;
    deflateInit(&def_str, Z_IPP_FAST_COMPRESSION);
    deflateCreateStat(&def_str);
    def_str.next_out = buffer_out;
    def_str.avail_out = sizeof(buffer_out);

    def_str.next_in = buffer1;
    def_str.avail_in = sizeof(buffer1);
    deflate(&def_str, Z_NO_FLUSH);

    def_str.next_in = buffer2;
    def_str.avail_in = sizeof(buffer2);
    deflate(&def_str, Z_FULL_FLUSH);
    /* Using of buffer_out is possible,
       it contains data compressed with fastest level,
       though without trained table */

    /* Generate and save trained table */
    tab_len = deflateGetTabLen(&def_str);
    p_table = malloc(tab_len);
    deflateGetTab(&def_str, p_table);

    /* Close statistics stream */
    deflateEnd(&def_str);

    /* Open a new stream for compression with trained table */
    table_str.zalloc = Z_NULL;
    table_str.zfree = Z_NULL;
    deflateInit(&table_str, Z_IPP_FAST_COMPRESSION);
    deflateUseTab(&table_str, p_table, tab_len);

    /* Then use table_str for compression as usually */

The definition of WITH_IPP and WITH_IPP_TABLES is important in the example above, because patched zlib.h file contains some extensions in conditional compilation constructions.

You can train deflate tables for different kind of data - text, binary, HTMLs - and re-use them on appropriate data.

Note
Important
Trained deflate table is an experimental mode in Intel® IPP Zlib, so use this feature with some caution.
Note
Don’t pass much data through statistics accumulation
There is no automatic checking of data statistics, so if too much data is compressed in the "statistics accumulation" mode, the statistic array overflow may happen. In most cases about 1 MB of input data is enough for good trained table generation.
...
    deflateInit(&def_str, Z_IPP_FAST_COMPRESSION);
    deflateCreateStat(&def_str);
...
    while(def_str.total_in < (1024 * 1024))
    {
        /* deflate... */
    }
    /* Generate and save trained table */
    tab_len = deflateGetTabLen(&def_str);
    p_table = malloc(tab_len);
    deflateGetTab(&def_str, p_table);
Note
Don’t use trained deflate table for inappropriate input data
Data compression is a data-driven process, so, in some cases, mixing of trained table for some kind of data with compression of different kind of data may decrease the compression ratio, in some cases the ratio could be even less than 1.
    deflateUseTab(&table_str, text_table_ptr, text_table_len);
    ...
    table_str.next_in = buffer_for_binary_data;
    ...
    deflate(&table_str, ...)    /* Don't do this! */

For example, below is table with compression ratio of Calgary* files, compressed with table trained on all Calgary* files (Calgary*-generated table).

File Default Ratio Ratio with File-generated Table Ratio with Calgary*-generated table

bib

2.15

2.60

2.52

book1

1.71

2.19

2.13

book2

2.06

2.54

2.48

geo

1.10

1.46

1.16

news

1.97

2.37

2.31

obj1

1.77

1.97

1.67

obj2

2.31

2.59

2.43

paper1

2.06

2.50

2.43

paper2

1.93

2.42

2.36

paper3

1.84

2.30

2.24

paper4

1.77

2.23

2.10

paper5

1.81

2.23

2.10

paper6

2.08

2.52

2.44

pic

6.76

8.28

6.75

progc

2.19

2.62

2.52

progl

3.03

3.63

3.50

progp

3.13

3.68

3.49

trans

3.45

3.98

3.81

You see that, for example, for the pic file the compression ratio with Calgary* generated table is worse than if no tables are used at all.

How to Build Zlib with Intel® Integrated Performance Primitives Library for Trained Deflate Tables

In order to build Zlib library with Intel® IPP functionality for trained deflate table you may use the procedure described in corresponding How to Build Zlib with Intel® Integrated Performance Primitives Library chapter. The only difference is, that you need to specify additional definition -DWITH_IPP_TABLES together with definition -DWITH_IPP.

So, correct compiler options for Linux* must be

$ export CFLAGS="-DWITH_IPP -DWITH_IPP_TABLES -I$IPPROOT/include"

On Windows* you need only to add a new definition of WITH_IPP_TABLES, but also add four new functions as exported symbols of dynamic library.

Static Intel® IPP linking:

> nmake -f win32\Makefile.msc LOC="-DWITH_IPP -DWITH_IPP_TABLES -I\"%IPPROOT%\include\"" ^
       LDFLAGS="-nologo -incremental:no -opt:ref ^
       /EXPORT:zzdeflateCreateStat /EXPORT:zzdeflateGetTabLen /EXPORT:zzdeflateGetTab ^
       /EXPORT:zzdeflateUseTab ^
       /LIBPATH:\"%IPPROOT%\lib\ia32\" ^
       ippdcmt.lib ippsmt.lib ippcoremt.lib"

Dynamic Intel® IPP linking:

> nmake -f win32\Makefile.msc LOC="-DWITH_IPP -DWITH_IPP_TABLES -I\"%IPPROOT%\include\"" ^
       LDFLAGS="-nologo -incremental:no -opt:ref ^
       /EXPORT:zzdeflateCreateStat /EXPORT:zzdeflateGetTabLen /EXPORT:zzdeflateGetTab ^
       /EXPORT:zzdeflateUseTab ^
       /LIBPATH:\"%IPPROOT%\lib\ia32\" ^
       ippdc.lib ipps.lib ippcore.lib"

Technical Support

If you did not register your Intel® software product during installation, please do so now at the Intel® Software Development Products Registration Center. Registration entitles you to free technical support, product updates and upgrades for the duration of the support term.

For general information about Intel technical support, product updates, user forums, FAQs, tips and tricks, and other support questions, please visit (http://www.intel.com/software/products/support).

Note
If your distributor provides technical support for this product, please contact them rather than Intel.

For technical information about the Intel® IPP library, including FAQ’s, tips and tricks, and other support information, please visit the Intel® IPP forum: (http://software.intel.com/en-us/forums/intel-integrated-performance-primitives) and browse the Intel® IPP support page: https://software.intel.com/en-us/intel-ipp-support/.