Disclaimer
Legal Information
INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS.
NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL
PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S
TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY
WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING
TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES
RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT
OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.
A "Mission Critical Application" is any application in which failure of the Intel Product could result,
directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY
SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES,
SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL
CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY,
ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL
APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING
OF THE INTEL PRODUCT OR ANY OF ITS PARTS.
Intel may make changes to specifications and product descriptions at any time,
without notice. Designers must not rely on the absence or characteristics of any
features or instructions marked "reserved" or "undefined." Intel reserves these
for future definition and shall have no responsibility whatsoever for conflicts
or incompatibilities arising from future changes to them. The information here
is subject to change without notice. Do not finalize a design with this
information.
The products described in this document may contain design defects or errors
known as errata which may cause the product to deviate from published
specifications. Current characterized errata are available on request.
Contact your local Intel sales office or your distributor to obtain the latest
specifications and before placing your product order.
Copies of documents which have an order number and are referenced in this
document, or other Intel literature, may be obtained by calling 1-800-548-4725,
or go to:
http://www.intel.com/design/literature.htm
Intel processor numbers are not a measure of performance. Processor numbers differentiate features within each processor family, not across different processor families. Go to: http://www.intel.com/products/processor_number/
BlueMoon, BunnyPeople, Celeron, Celeron Inside, Centrino, Centrino Inside, Core Inside,
i960, Intel, the Intel logo, Intel Atom, Intel Atom Inside, Intel Core,
Intel Inside, Intel Inside logo, Intel NetBurst, Intel NetMerge,
Intel NetStructure, Intel SingleDriver, Intel SpeedStep,
Intel Sponsors of Tomorrow., the Intel Sponsors of Tomorrow. logo,
Intel StrataFlash, Intel Viiv, Intel vPro, Intel XScale, InTru, the InTru logo,
InTru soundmark, Itanium, Itanium Inside, MCS, MMX, Moblin, Pentium,
Pentium Inside, skoool, the skoool logo, Sound Mark, The Journey Inside,
vPro Inside, VTune, Xeon, and Xeon Inside
are trademarks of Intel Corporation in the U.S. and other countries.
* Other names and brands may be claimed as the property of others.
Microsoft, Windows, Visual Studio, Visual C++, and the Windows logo are trademarks, or registered trademarks of Microsoft Corporation in the United States and/or other countries.
Java is a registered trademark of Oracle and/or its affiliates.
©Intel Corporation. All rights reserved.
Optimization Notice |
---|
Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not
unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and
other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of
any optimization on microprocessors not manufactured by Intel.
Notice revision #20110804 |
License Definitions
By downloading and installing this product, you hereby agree that the accompanying materials are being provided to you under the terms and conditions of the End User License Agreement for the Intel® Integrated Performance Primitives (Intel® IPP) product previously accepted by you.
System Requirements
Hardware requirements:
-
A system based on Intel® Pentium® processor, Intel® Xeon® processor, or subsequent IA-32 architecture based system;
-
A system based on a processor with Intel® 64 architecture.
Software requirements:
-
For the basic system requirements on operating system, Intel® IPP library, IDE and compiler, refer to the corresponding Release Notes document.
For more information please see https://software.intel.com/en-us/ipp [Intel® Integrated Performance Primitives].
What’s New
The following updates have been made to Intel® IPP functions for Zlib:
-
Removed source code patch file for Zlib version 1.2.5.3. If you want to speed up this particular version of Zlib, please use patch file from previous distributions of Intel® IPP
-
Significantly improved performance of the data compression functions without degradation of the compression ratio
-
Implemented a new fastest data compression mode. The compression ratio of the new mode is a bit less than the fast mode’s ratio - compression level 1, but performance is significantly better. To use this mode, specify the negative compression level equal to -2 (since -1 is used for default compression level in Zlib) when calling the
deflateInit
function:
deflateInit(&deflate_stream, -2);
You can use symbolic name Z_IPP_FAST_COMPRESSION
, which is equal to -2:
deflateInit(&deflate_stream, Z_IPP_FAST_COMPRESSION);
-
Introduced new sub-ranges of compression levels from 11 to 29
-
The standard levels 1..9 are compatible with original Zlib
-
Levels 11..19 are similar to 1..9, but fit better when compressing large files (more than 1 MB)
-
Levels 21..29 are for highly compressible files with compression ratio equal to 30x and higher.
-
-
Activated Zlib
deflateTune
function. The following is quote from Zlib programming manual:
ZEXTERN int ZEXPORT deflateTune OF((z_streamp strm,
int good_length,
int max_lazy,
int nice_length,
int max_chain));
Fine tune deflate's internal compression parameters. This should only be used by
someone who understands the algorithm used by Zlib's +deflate+ for searching for the
best matching string, and even then only by the most fanatic optimizer trying
to squeeze out the last compressed bit for their specific input data.
Read the deflate.c source code for the meaning of the max_lazy, good_length,
nice_length, and max_chain parameters.
The described function arguments now affect Intel® IPP data compression functions.
You can find the default sets of string matching parameters in the
configuration_table
array inside deflate.c
source code file.
-
Added the new "trained deflate tables" feature for
Z_IPP_FAST_COMPRESSION
- fastest compression mode (see New Intel® IPP Functions for Compression with Trained Deflate Tables).
How to Build Zlib with Intel® Integrated Performance Primitives Library
The data compression domain of the Intel® IPP library contains several functions that can speed up the well-known Zlib library (http://zlib.net) in both data compression and decompression operations. For the detailed description of these functions, refer to the Volume 1: "Signal Processing/Data Compression Functions/Dictionary-Based Compression Functions/ZLIB Coding Functions" section of the Intel® IPP Developer Reference.
To use these functions, before building the library you need to replace the original source code of several Zlib functions with Intel® IPP functions calls.
This document provides instructions on how to prepare the Zlib source code and build the library.
Prerequisites
The Zlib repository contains several stable versions of the library: v.1.2.6.1, v.1.2.7.3, v.1.2.8 and v.1.2.11. You may want to update and speed up a particular Zlib version, which is used in your application, or on your operating system. For that, Intel® IPP product provides four source code patch files applicable to specific versions of Zlib.
Downloading, Patching and Build on Linux* and macOS*
Downloading Zlib Source Code
Before building the library, you need to download Zlib source code files from the Zlib site (http://zlib.net). The following table provides links to specific Zlib versions that can be updated with Intel® IPP library function calls.
Zlib version | Web address |
---|---|
|
|
|
|
|
|
|
Download the tar file from the specified address to a working directory. Unpack the downloaded archive with the following command on Linux*/macOS* systems.
$ tar xvf <archive file name>
This command will create the zlib-<version>
folder in your
working directory, where <version>
is selected Zlib version number from 1.2.6.1
to 1.2.11.
Patching the Source Code
-
Change directory to the
zlib-<version>
folder. -
Apply the corresponding source code patch file
zlib-<version>.patch.bin
from Intel® IPP product distribution with the following command
$ patch -p1 < "path to corresponding patch file"
For example, to patch Zlib v.1.2.6.1, go to zlib-1.2.6.1
directory and run
the following command
$ patch -p1 < ../zlib-1.2.6.1.patch.bin
If the patching process completed successfully, you will see the following messages:
patching file adler32.c
patching file crc32.c
patching file deflate.c
patching file inflate.c
patching file inftrees.h
patching file trees.c
If something goes wrong, for example, if the adler32.c
file has been changed
(nonoriginal), you will see the diagnostics messages:
patching file adler32.c
Hunk #1 FAILED at 5.
patching file crc32.c
patching file deflate.c
patching file inflate.c
patching file inftrees.h
patching file trees.c
Configuring and Building Zlib Library
After Zlib source code files are patched successfully, you need to create a makefile for building process. Basically, you need to
-
Add the
WITH_IPP
definition to compiler command line -
Add the
$IPPROOT/include
directory to the list of header files search for compiler -
Add Intel® IPP libraries to the list of linker input files.
On Linux* OS, you can implement the above steps with the standard Zlib configure/make sequence and the following commands (32-bit example):
# Build dynamic Zlib library with static Intel(R) IPP linkage on Linux*
$ export CFLAGS="-m32 -DWITH_IPP -I$IPPROOT/include"
$ export LDFLAGS="$IPPROOT/lib/ia32/libippdc.a $IPPROOT/lib/ia32/libipps.a $IPPROOT/lib/ia32/libippcore.a"
$ ./configure
$ make shared
Here, the -m32
option specifies IA-32 target CPU architecture. For Intel® 64
architecture the compiler option should be -m64
.
The above commands assume that IPPROOT
environment variable points to
head of Intel® IPP location.
Configure your Intel® IPP working environment according to "Get Started with Intel® Integrated Performance Primitives" document.
Note
|
The conditional compilation statements, used for Zlib source code files modification,
allow to build original version of Zlib from the same source code files. For that,
you need to build Zlib library without modification of CFLAGS and LDFLAGS .
If WITH_IPP definition doesn’t exist in compiler options, Zlib source code files
are compiled as unpatched. |
Static and Dynamic Linking to Intel® IPP Libraries
When building the libz
dynamic shared object, you can use either static or
dynamic linking to Intel® IPP libraries. Different linking modes have their
advantages and disadvantages. For example, static linking to Intel® IPP
provides target object (libz.so
, or libz.dylib
in our case) independence
from environment, easy transfer from one computer to another. On the other hand,
statically linked object has bigger size and can’t benefit from newer (updated)
version of Intel® IPP without re-linking.
The libz
dynamic libraries, linked statically with Intel® IPP, are
self-sufficient. It means that when you build an application using these
libraries, you don’t need to add Intel® IPP libraries to the linker command
line (unless your application uses Intel® IPP apart from libz
).
All references to Intel® IPP functions are already resolved, and required
Intel® IPP functions code is inside the libz
dynamic library.
Build Zlib with Static Linking to Intel® IPP
On Linux* OS, when the below linker flags are specified by the LDFLAGS
environment variable, Zlib is linked to Intel® IPP libraries (.a files)
statically:
$ export LDFLAGS="$IPPROOT/lib/intel64/libippdc.a $IPPROOT/lib/intel64/libipps.a \
$IPPROOT/lib/intel64/libippcore.a"
You can check the independence of the resulting dynamic library libz.so
from Intel® IPP libraries with the "ldd" tool.
For example,
$ ldd libz.so
linux-vdso.so.1 => (0x00007fff885fe000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb169e38000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb16a503000)
You can see that the libz.so
dynamic library depends on system dynamic
libraries only. The size of the resulting libz
library in this case will be more
than 700 KBytes because it contains optimizations for all CPUs supported by
Intel® IPP.
On macOS*, the similar tool is called "otool". For example,
$ otool -L libz.dylib
libz.dylib:
/usr/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
You can use the following command to build an application with such kind of library:
$ cc -o zlib_test -Izlib/include zlib_test.c -L. -lz
Here, zlib/include
is a directory where zlib header files reside
(in most cases zlib.h
and zconf.h
are required), "-L." means that Linux*
linker will be searching next libraries in the current directory, "-lz" means
that the linker will look for the libz.so
file (shared linking is default
method on most Linux* systems).
To run built application on Linux* system, you need to modify the
LD_LIBRARY_PATH
environment variable, which describes the search path for
dynamic libraries:
$ export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH
$ ./zlib_test
On macOS* systems the corresponding environment variable is DYLD_LIBRARY_PATH
:
$ export DYLD_LIBRARY_PATH=.:$DYLD_LIBRARY_PATH
$ ./zlib_test
Building Zlib with Dynamic Linking to Intel® IPP
Sometimes, you may want to build libz
dynamic library linked dynamically to
Intel® IPP libraries. This is, for example, if you know that libz
will be
used on a computer with pre-installed and properly set dynamic Intel® IPP
libraries. Or, you may want to minimize file size of libz
dynamic library,
or, to decrease the total amount of RAM size required for several application(s)
that use libz
.
In this case, the following set of commands will build the libz.so
library with
dynamic linking to the Intel® IPP libraries (64-bit example).
$ <Configure Intel(R) IPP environment>
$ export CFLAGS="-m64 -DWITH_IPP -I$IPPROOT/include"
$ export LDFLAGS="-L$IPPROOT/lib/intel64 -lippdc -lipps -lippcore"
$ ./configure
$ make shared
On macOS* system, the corresponding set of command will be:
$ <Configure Intel(R) IPP environment>
$ export CFLAGS="-m64 -DWITH_IPP -I$IPPROOT/include"
$ export LDFLAGS="-L$IPPROOT/lib -lippdc -lipps -lippcore"
$ ./configure
$ make shared
$ otool -L libz.dylib
On a target computer, before the libz
library is used, you need to properly
set paths to Intel® IPP dynamic libraries. On Linux* OS, do this by
modifying the LD_LIBRARY_PATH
environment variable. On macOS*, modify
DYLD_LIBRARY_PATH
. If you use compilervars
script to set working environment,
these environment variables are set automatically.
Using Static libz.a
Library to Build User Applications
During standard "configure/make" process the makefile creates the libz.a
static library with libz
compiled functions. You may want to use this static
library with your application instead of using the libz
shared object.
The static libz.a
library contains unresolved external references to
Intel® IPP library functions, so, when you build your application with the
static libz.a
library, you still need to solve the problem of linking to
Intel® IPP libraries.
For static linking to Intel® IPP, use the following linking options:
libz.a $IPPROOT/lib/<arch>/libippdc.a $IPPROOT/lib/<arch>/libipps.a $IPPROOT/lib/<arch>/libippcore.a
where <arch> is either ia32
or intel64
depending on kind of application
(32-bit or 64-bit) you are going to build.
On macOS* the corresponding linking options are
libz.a $IPPROOT/lib/libippdc.a $IPPROOT/lib/libipps.a $IPPROOT/lib/libippcore.a
For dynamic linking to Intel® IPP on Linux* OS, use the following options:
libz.a -L$IPPROOT/lib/<arch> -lippdc -lipps -lippcore
On macOS*
libz.a -L$IPPROOT/lib -lippdc -lipps -lippcore
Downloading, Patching, and Build on Windows*
To download Zlib source code files for Windows*, you can use the same URL addresses as for Linux*. However, the patching and building processes on Windows* are different.
Unpacking Zlib Tarball
To get the Zlib source code files on Windows* computer, you need to:
-
Download the
.tar.gz
file from http://zlib.net -
Unpack the archive using any of available tar tools for Windows*. For example
-
Pre-built
tar
from Free Software Foundation (http://gnuwin32.sourceforge.net/packages/gtar.htm) -
7-zip
from http://www.7-zip.org/.
-
Patching Original Zlib Files
There are no standard system tools for patching on Windows*, so you need to use 3-rd party tools. For example, http://gnuwin32.sourceforge.net/packages/patch.htm from Free Software Foundation.
With this tool you need to use the following command line options:
> patch.exe -p1 --binary < <patch file location>
"--binary" command line option here is important to avoid confusion because of Linux*/Windows* "end-of-line" character difference.
Example for zlib-1.2.7.3 is below:
> tar.exe xvf zlib-1.2.7.3.tar.gz
> cd zlib-1.2.7.3
> patch.exe -p1 --binary < ..\zlib-1.2.7.3.patch.bin
patching file adler32.c
patching file crc32.c
patching file deflate.c
patching file inflate.c
patching file inftrees.h
patching file trees.c
Building Zlib With Intel® IPP on Windows*
To build Zlib on Windows*, use the "nmake" file Makefile.msc
from the zlib\win32
directory. To do that, start Microsoft* Visual Studio* command-line window that
corresponds to a target application architecture (IA-32 or Intel® 64) and
execute the following command from the root of Zlib source code tree
(below are 32-bit examples).
Static linking:
> nmake -f win32\Makefile.msc LOC="-DWITH_IPP -I\"%IPPROOT%\include\"" ^
LDFLAGS="-nologo -incremental:no ^
-opt:ref /LIBPATH:\"%IPPROOT%\lib\ia32\" ^
ippdcmt.lib ippsmt.lib ippcoremt.lib"
Dynamic linking:
> nmake -f win32\Makefile.msc LOC="-DWITH_IPP -I\"%IPPROOT%\include\"" ^
LDFLAGS="-nologo -incremental:no ^
-opt:ref /LIBPATH:\"%IPPROOT%\lib\ia32\" ^
ippdc.lib ipps.lib ippcore.lib"
In the first case the "nmake" creates the zlib1.dll
file not depending on
dynamic Intel® IPP libraries. In the second, zlib1.dll
depends on
ippdc.dll
, ipps.dll
and ippcore.dll
dynamic libraries.
Building User Application with the Static zlib.lib
library
When static linking is used, the Zlib build procedure, along with zlib1.dll
,
creates the static Zlib library zlib.lib
. It can be used with application
object files to be built statically.
If you want the Intel® IPP library to be linked to your application statically, you need to specify the following input files in the command line to Microsoft* linker:
"%IPPROOT%\lib\<arch>\ippdcmt.lib" "%IPPROOT%\lib\<arch>\ippsmt.lib" "%IPPROOT%\lib\<arch>\ippcoremt.lib"
For dynamic linking, you need to choose other libraries from the Intel® IPP library directory:
"%IPPROOT%\lib\<arch>\ippdc.lib" "%IPPROOT%\lib\<arch>\ipps.lib" "%IPPROOT%\lib\<arch>\ippcore.lib"
These libraries are just references to Intel® IPP dynamic libraries (DLLs).
Note
|
For Windows* build procedure it is important to use double quotes """ around
%IPPROOT% , because by default Intel® IPP is installed to the
"Program Files(x86) " directory. Its name contains spaces and, without quotes
around, this name can confuse Microsoft* compiler and/or linker. |
How to Verify Zlib with Intel® Integrated Performance Primitives Library
For smoke testing or verification of the newly built library, you can use small tests/examples, which are supplied with the Zlib distribution.
Verification on Linux* and macOS*
The makefile generated by the configure
utility can be used to test library
build with Makefile’s test
target:
$ make test
Dynamic libz
Library and Dynamic Tests
In general, command lines to build and test are (64-bit Linux* example on zlib 1.2.11):
$ export CFLAGS="-m64 -DWITH_IPP -I$IPPROOT/include"
$ export LDFLAGS="$IPPROOT/lib/intel64/libippdc.a \
$IPPROOT/lib/intel64/libipps.a $IPPROOT/lib/intel64/libippcore.a"
$ ./configure
$ make testshared
hello world
zlib version 1.2.11 = 0x12b0, compile flags = 0xa9
uncompress(): hello, hello!
gzread(): hello, hello!
gzgets() after gzseek: hello!
inflate(): hello, hello!
large_inflate(): OK
after inflateSync(): hello, hello!
inflate with dictionary: hello, hello!
*** zlib shared test OK ***
Static libz
Library and Static Tests
For static tests you need to use slightly different approach, because in
original Zlib makefile LDFLAGS
variable is not used for static build.
TEST_LDFLAGS
argument in "make" command line must be used instead.
So, the command lines should be (32-bit Linux* example)
$ export CFLAGS="-m32 -DWITH_IPP -I$IPPROOT/include"
$ ./configure
$ make teststatic TEST_LDFLAGS="libz.a $IPPROOT/lib/ia32/libippdc.a \
$IPPROOT/lib/ia32/libipps.a $IPPROOT/lib/ia32/libippcore.a"
hello world
zlib version 1.2.11 = 0x12b0, compile flags = 0x55
uncompress(): hello, hello!
gzread(): hello, hello!
gzgets() after gzseek: hello!
inflate(): hello, hello!
large_inflate(): OK
after inflateSync(): hello, hello!
inflate with dictionary: hello, hello!
*** zlib test OK ***
Test libz
on macOS*
Below are the same example command lines for macOS*
$ export CFLAGS="-m64 -DWITH_IPP -I$IPPROOT/include"
$ export LDFLAGS="$IPPROOT/lib/libippdc.a $IPPROOT/lib/libipps.a \
$IPPROOT/lib/libippcore.a"
$ ./configure
$ make testshared
hello world
zlib version 1.2.8 = 0x1280, compile flags = 0xa9
uncompress(): hello, hello!
gzread(): hello, hello!
gzgets() after gzseek: hello!
inflate(): hello, hello!
large_inflate(): OK
after inflateSync(): hello, hello!
inflate with dictionary: hello, hello!
*** zlib shared test OK ***
and
$ export CFLAGS="-m32 -DWITH_IPP -I$IPPROOT/include"
$ ./configure
$ make teststatic TEST_LDFLAGS="libz.a $IPPROOT/lib/libippdc.a \
$IPPROOT/lib/libipps.a $IPPROOT/lib/libippcore.a"
hello world
zlib version 1.2.8 = 0x1280, compile flags = 0x95
uncompress(): hello, hello!
gzread(): hello, hello!
gzgets() after gzseek: hello!
inflate(): hello, hello!
large_inflate(): OK
after inflateSync(): hello, hello!
inflate with dictionary: hello, hello!
*** zlib test OK ***
Verification on Windows*
Command line to build and test dynamic Zlib library on Windows* is
nmake -f win32\Makefile.msc LOC="-DWITH_IPP -I\"%IPPROOT%\include\"" ^
LDFLAGS="-nologo -incremental :no -opt:ref ^
/LIBPATH:\"%IPPROOT%\lib\ia32\" ^
ippdcmt.lib ippsmt.lib ippcoremt.lib" testdll
Microsoft (R) Program Maintenance Utility Version 14.00.23506.0
Copyright (C) Microsoft Corporation. All rights reserved.
example_d
zlib version 1.2.6.1 = 0x1261, compile flags = 0x55
uncompress(): hello, hello!
gzread(): hello, hello!
gzgets() after gzseek: hello!
inflate(): hello, hello!
large_inflate(): OK
after inflateSync(): hello, hello!
inflate with dictionary: hello, hello!
echo hello world | minigzip_d | minigzip_d -d
hello world
Using of Trained Deflate Tables in Fastest Compression Mode
In Intel® IPP version of Zlib, the fastest compression mode is performed using
the simplified pattern of the search algorithm and special table for Huffman compression.
By default, the static Huffman compression table is used.
This table provides an average compression ratio suitable for all kinds of deflate
input data.
If deflate
is used to compress more or less uniform input data, for example texts,
HTML files, or XML tables, you can significantly improve the compression ratio
without losses to compression performance by using specific compression tables,
which contain pre-built statistical data on such kind of information.
To generate and use trained deflate tables, the following functionality is added to the original Zlib distribution:
#if defined(WITH_IPP) && defined(WITH_IPP_TABLES) ZEXTERN int ZEXPORT zzdeflateCreateStat OF((z_streamp)); ZEXTERN int ZEXPORT zzdeflateGetTabLen OF((z_streamp)); ZEXTERN int ZEXPORT zzdeflateGetTab OF((z_streamp, voidpf)); ZEXTERN int ZEXPORT zzdeflateUseTab OF((z_streamp, voidpf, int)); #define deflateCreateStat zzdeflateCreateStat #define deflateGetTabLen zzdeflateGetTabLen #define deflateGetTab zzdeflateGetTab #define deflateUseTab zzdeflateUseTab #endif
New Intel® IPP Functions for Compression with Trained Deflate Tables
The new functionality is implemented in the scope of original z_stream
objects.
To use functions for statistics collection, table generation and table use, you need
to have initialized z_stream
objects.
Function names start with "zz" prefix and the functions are placed at the
end of dynamic libraries (dynamic object .so on Linux* and DLL on Windows*).
This prevents from breaking ordinal numbers of functions in dynamic library, though
this usage model - ordinal numbers instead of function names - is not recommended by
Zlib documentation. In zlib.h
file the new function names are also redefined to make
them more useful.
deflateCreateStat
Function
Syntax:
int deflateCreateStat(z_streamp stream_ptr);
This function starts collecting of input data statistics for all subsequent calls
to deflate
function on the same stream. The deflateCreateStat
function must be
used with address of initialized stream as argument. The typical use is
int zstatus; z_stream stat_stream; stat_stream.zalloc = Z_NULL; stat_stream.zfree = Z_NULL; zstatus = deflateInit(&stat_stream, Z_IPP_FAST_COMPRESSION); zstatus = deflateCreateStat(&stat_stream);
After initialization all calls to deflate
function, besides performing
compression itself, accumulate input data entropy statistics.
The return values are:
Value | Description |
---|---|
|
Success |
|
Argument is |
|
Function is not able to allocate internal arrays |
deflateGetTabLen
Functon
Syntax:
int deflateGetTabLen(z_streamp stream_ptr);
This function stops accumulating of input data statistics, calculates the size
of trained table, generates the table and returns its length to calling application.
The table is stored internally z_stream
until deflateGetTab
function is called.
The function accepts pointer to z_stream
with accumulated statistics as its argument.
Return value is length of generated table. Application must allocate dynamic memory
buffer of that length (usually, length is about 1K bytes) and, then must call
deflateGetTab
function to copy generated table from z_stream
to external buffer
for subsequent use.
Return values are:
Value | Description |
---|---|
> 0 |
Size of generated table |
|
Argument is |
0 |
Internal Intel® IPP function failed |
deflateGetTab
Function
Syntax:
int deflateGetTab(z_streamp stream_ptr, voidpf table_ptr);
This function copies generated deflate table from z_stream
internal buffer to
buffer allocated in application memory for external use. The arguments are: s_streamp
pointer to z_stream
with generated table, voidpf
is a void*
pointer to external buffer.
The typical use of deflateGetTabLen
and deflateGetTab
function is:
int tab_len; void* tab_ptr; int zstatus; /* Accimulate statistics for part of input data... */ tab_len = deflateGetTabLen(&statistic_stream); tab_ptr = malloc(tab_len); zstatus = deflateGetTab(&statistic_stream, tab_ptr);
The return values are:
Value | Description |
---|---|
|
Success |
|
Argument is |
deflateUseTab
Function
Syntax:
int deflateUseTab(z_streamp stream_ptr, voidpf table_ptr, int table_len);
This function copies buffer with trained table to z_stream
internal memory.
During the next deflate
function calls for that stream, this trained table
will be used to increase ratio of the compression operations.
Return values are:
Value | Description |
---|---|
|
Success |
|
Argument is |
The operations with z_stream
, associated with trained table, are as usual.
Note
|
Currently, z_stream object is not designed to change trained deflate tables
"on-the-run", so to apply another trained table to compress other kind of input data, you need
to close stream with previous table and open a new stream for another trained table. |
Compression Ratio Results of Intel® IPP Zlib with Trained Deflate Tables
The table below shows the compression ratio and performance of deflating of files from Calgary* data compression corpus without- and with trained compression table(s).
-
CPU: Intel® Core™ i5-4300U @ 2.5 GHz on Windows* 8.1 Enterprise, 64 bits
-
Compression ratio is calculated as <file size before compression> / <file size after compression>
-
Compression performance is <file size before compression> / <compression time> in megabytes per second
-
Compression level is -2 (
Z_IPP_FAST_COMPRESSION
).
File | Default Ratio | Default Compression Performance | Ratio with Table | Performance with Table |
---|---|---|---|---|
bib |
2.15 |
176 |
2.60 |
179 |
book1 |
1.71 |
126 |
2.19 |
158 |
book2 |
2.06 |
177 |
2.54 |
186 |
geo |
1.10 |
93 |
1.46 |
96 |
news |
1.97 |
130 |
2.37 |
148 |
obj1 |
1.77 |
206 |
1.97 |
222 |
obj2 |
2.31 |
178 |
2.59 |
177 |
paper1 |
2.06 |
179 |
2.50 |
186 |
paper2 |
1.93 |
157 |
2.42 |
172 |
paper3 |
1.84 |
159 |
2.30 |
186 |
paper4 |
1.77 |
177 |
2.23 |
240 |
paper5 |
1.81 |
176 |
2.23 |
246 |
paper6 |
2.08 |
200 |
2.52 |
214 |
pic |
6.76 |
534 |
8.28 |
519 |
progc |
2.19 |
203 |
2.62 |
211 |
progl |
3.03 |
245 |
3.63 |
252 |
progp |
3.13 |
273 |
3.68 |
288 |
trans |
3.45 |
307 |
3.98 |
309 |
The data above shows the best result when a separate table is generated for each Calgary* file and this table is used for compression. If you generate a trained table for a set of files, the resulting compression ratio will be less by some points. The compression performance will likely be the same.
How to Create and Use Trained Deflate Tables
Before you start using a trained deflate table, you need to build it. The build process includes two phases:
1) Accumulation of statistics on input data
2) Generation of the trained table.
Accumulation of statistics can be done in parallel with ordinary compression.
For example,
#define WITH_IPP #define WITH_IPP_TABLES #include "zlib.h" ... z_stream def_str; z_stream table_str; int tab_len; void* p_table; def_str.zalloc = Z_NULL; def_str.zfree = Z_NULL; deflateInit(&def_str, Z_IPP_FAST_COMPRESSION); deflateCreateStat(&def_str); def_str.next_out = buffer_out; def_str.avail_out = sizeof(buffer_out); def_str.next_in = buffer1; def_str.avail_in = sizeof(buffer1); deflate(&def_str, Z_NO_FLUSH); def_str.next_in = buffer2; def_str.avail_in = sizeof(buffer2); deflate(&def_str, Z_FULL_FLUSH); /* Using of buffer_out is possible, it contains data compressed with fastest level, though without trained table */ /* Generate and save trained table */ tab_len = deflateGetTabLen(&def_str); p_table = malloc(tab_len); deflateGetTab(&def_str, p_table); /* Close statistics stream */ deflateEnd(&def_str); /* Open a new stream for compression with trained table */ table_str.zalloc = Z_NULL; table_str.zfree = Z_NULL; deflateInit(&table_str, Z_IPP_FAST_COMPRESSION); deflateUseTab(&table_str, p_table, tab_len); /* Then use table_str for compression as usually */
The definition of WITH_IPP
and WITH_IPP_TABLES
is important in the example above,
because patched zlib.h
file contains some extensions in conditional compilation
constructions.
You can train deflate tables for different kind of data - text, binary, HTMLs - and re-use them on appropriate data.
Note
|
Important Trained deflate table is an experimental mode in Intel® IPP Zlib, so use this
feature with some caution. |
Note
|
Don’t pass much data through statistics accumulation There is no automatic checking of data statistics, so if too much data is
compressed in the "statistics accumulation" mode, the statistic array
overflow may happen. In most cases about 1 MB of input data is enough for
good trained table generation. |
... deflateInit(&def_str, Z_IPP_FAST_COMPRESSION); deflateCreateStat(&def_str); ... while(def_str.total_in < (1024 * 1024)) { /* deflate... */ } /* Generate and save trained table */ tab_len = deflateGetTabLen(&def_str); p_table = malloc(tab_len); deflateGetTab(&def_str, p_table);
Note
|
Don’t use trained deflate table for inappropriate input data Data compression is a data-driven process, so, in some cases, mixing of trained
table for some kind of data with compression of different kind of data may
decrease the compression ratio, in some cases the ratio could be even less than 1. |
deflateUseTab(&table_str, text_table_ptr, text_table_len); ... table_str.next_in = buffer_for_binary_data; ... deflate(&table_str, ...) /* Don't do this! */
For example, below is table with compression ratio of Calgary* files, compressed with table trained on all Calgary* files (Calgary*-generated table).
File | Default Ratio | Ratio with File-generated Table | Ratio with Calgary*-generated table |
---|---|---|---|
bib |
2.15 |
2.60 |
2.52 |
book1 |
1.71 |
2.19 |
2.13 |
book2 |
2.06 |
2.54 |
2.48 |
geo |
1.10 |
1.46 |
1.16 |
news |
1.97 |
2.37 |
2.31 |
obj1 |
1.77 |
1.97 |
1.67 |
obj2 |
2.31 |
2.59 |
2.43 |
paper1 |
2.06 |
2.50 |
2.43 |
paper2 |
1.93 |
2.42 |
2.36 |
paper3 |
1.84 |
2.30 |
2.24 |
paper4 |
1.77 |
2.23 |
2.10 |
paper5 |
1.81 |
2.23 |
2.10 |
paper6 |
2.08 |
2.52 |
2.44 |
pic |
6.76 |
8.28 |
6.75 |
progc |
2.19 |
2.62 |
2.52 |
progl |
3.03 |
3.63 |
3.50 |
progp |
3.13 |
3.68 |
3.49 |
trans |
3.45 |
3.98 |
3.81 |
You see that, for example, for the pic
file the compression ratio with Calgary*
generated table is worse than if no tables are used at all.
How to Build Zlib with Intel® Integrated Performance Primitives Library for Trained Deflate Tables
In order to build Zlib library with Intel® IPP functionality for trained deflate table
you may use the procedure described in corresponding
How to Build Zlib with Intel® Integrated Performance Primitives Library
chapter.
The only difference is, that you need to specify additional definition
-DWITH_IPP_TABLES
together with definition -DWITH_IPP
.
So, correct compiler options for Linux* must be
$ export CFLAGS="-DWITH_IPP -DWITH_IPP_TABLES -I$IPPROOT/include"
On Windows* you need only to add a new definition of WITH_IPP_TABLES
, but also
add four new functions as exported symbols of dynamic library.
Static Intel® IPP linking:
> nmake -f win32\Makefile.msc LOC="-DWITH_IPP -DWITH_IPP_TABLES -I\"%IPPROOT%\include\"" ^
LDFLAGS="-nologo -incremental:no -opt:ref ^
/EXPORT:zzdeflateCreateStat /EXPORT:zzdeflateGetTabLen /EXPORT:zzdeflateGetTab ^
/EXPORT:zzdeflateUseTab ^
/LIBPATH:\"%IPPROOT%\lib\ia32\" ^
ippdcmt.lib ippsmt.lib ippcoremt.lib"
Dynamic Intel® IPP linking:
> nmake -f win32\Makefile.msc LOC="-DWITH_IPP -DWITH_IPP_TABLES -I\"%IPPROOT%\include\"" ^
LDFLAGS="-nologo -incremental:no -opt:ref ^
/EXPORT:zzdeflateCreateStat /EXPORT:zzdeflateGetTabLen /EXPORT:zzdeflateGetTab ^
/EXPORT:zzdeflateUseTab ^
/LIBPATH:\"%IPPROOT%\lib\ia32\" ^
ippdc.lib ipps.lib ippcore.lib"
Technical Support
If you did not register your Intel® software product during installation, please do so now at the Intel® Software Development Products Registration Center. Registration entitles you to free technical support, product updates and upgrades for the duration of the support term.
For general information about Intel technical support, product updates, user forums, FAQs, tips and tricks, and other support questions, please visit (http://www.intel.com/software/products/support).
Note
|
If your distributor provides technical support for this product, please contact them rather than Intel. |
For technical information about the Intel® IPP library, including FAQ’s, tips and tricks, and other support information, please visit the Intel® IPP forum: (http://software.intel.com/en-us/forums/intel-integrated-performance-primitives) and browse the Intel® IPP support page: https://software.intel.com/en-us/intel-ipp-support/.