Package and Environment Managers
Python* package and environment managers are essential tools for productivity. These allow for installation, updating, and removal of no longer needed packages from the Python environment. They also manage multiple Python environments customized by a user for multiple purposes or needs.
|
Package |
Description |
References |
How to Get It |
|
conda* |
A popular package manager that comes with the Python installation. You do not need to install it separately. |
conda install conda -c intel conda install conda -c main Download Intel® Distribution for Python* Download Intel® AI Analytics Toolkit Access Intel® Developer Cloud |
|
|
pip |
A classic Python package manager that comes with the Python installation. You do not need to install it separately. |
conda install pip -c intel conda install pip -c main Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
Data Processing and Modeling
Data processing is an essential part of numerical and data science workflows, and is based on numerical and statistical algorithms. It includes data collection, ingestion, preprocessing, normalization, transformation, aggregation, and analysis. Data analysis serves a basis for modeling of phenomena.
|
Package |
Description |
References |
How to Get It |
|
numpy |
A foundational math library for core operations on N-dimensional arrays, optimized for Intel CPUs through Intel® oneAPI Math Kernel Library (oneMKL). Optimizations include linear algebra, fast Fourier transform (FFT), universal functions, and random number generators (RNG). |
|
conda install numpy -c intel Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
scipy |
A scientific library for numerical integration, interpolation, optimization, linear algebra, and statistics. The Intel-optimized version is based on oneMKL. |
|
conda install scipy -c intel Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
numexpr |
A lazy evaluation of numerical expressions for NumPy that optimizes memory use and makes use of all available CPU cores. The Intel-optimized version is based on oneMKL. |
|
conda install -c intel scipy Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
mkl_service |
Python interfaces to oneMKL service functions for controlling MKL behavior. It is not typically necessary to install this package with Intel optimized NumPy, SciPy, or NumExpr because they come with this package. However, it may be needed with other Python packages that rely on oneMKL if fine-grain control is necessary. |
|
conda install mkl-service -c intel Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
mkl_fft |
Python interfaces to oneMKL FFT functions. It is not typically necessary to install this package with Intel-optimized NumPy or SciPy because they come with this package. However, it may be needed to directly access MKL FFT interfaces rather than through NumPy or SciPy. |
|
conda install mkl_fft -c intel Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
mkl_umath |
Python interfaces to oneMKL vector math library (VML) functions. It is not typically necessary to install this package with Intel-optimized NumPy or NumExpr because they come with this package. However, it may be needed to directly access MKL VML interfaces rather than through NumPy or NumExpr. |
|
conda install mkl_umath -c intel Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
mkl_random |
Python interfaces to oneMKL RNG functions. It is not typically necessary to install this package with Intel-optimized NumPy because it comes with this package. However, it may be needed to directly access MKL RNG interfaces rather than through NumPy. |
|
conda install mkl_random -c intel Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
dpnp |
The Data Parallel Extensions for NumPy library implements a subset of NumPy that can be offloaded to Intel GPUs. Learn more about Data Parallel Extensions for Python. |
|
conda install dpnp -c intel Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
Foundational Machine Learning
Deep learning and machine learning are special models that allow a machine to automatically learn from data without programming it explicitly.
While Intel Distribution for Python includes a few foundational machine learning packages, Intel AI Analytics Toolkit provides a more comprehensive set of tools for deep learning and machine learning.
|
Package |
Description |
Reference |
How to Get It |
|
scikit-learn |
A comprehensive set of machine learning algorithms for classification, regression, clustering, dimensionality reduction, model selection, and preprocessing. |
|
conda install scikit-learn -c intelpip install Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
scikit-learn-intelex |
Intel® Extension for Scikit-learn* provides scikit-learn like interfaces for Intel® oneAPI Data Analytics Library (oneDAL). It is not typically necessary to install Intel Extension for Scikit-learn as it comes with Intel optimized scikit-learn. You can install the extension with standard scikit-learn to enable Intel optimizations and additional capabilities. |
|
conda install scikit-learn-intelex -c intel pip install scikit-learn-intelex Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
xgboost |
XGBoost is a regularizing gradient boosting framework. |
|
conda install xgboost -c intel pip install xgboost Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
Python Interpreter and Compilers
The Python interpreter is the core of a versatile interactive experience with the language. It is expressive, intuitive, and fast enough for interactive usage. It is robust and secure for enterprise installations.
Python compilers are an important addition to the Python interpreter for scaling up Python applications. While initial prototyping work is mostly interactive, in a production setting the Python interpreter may become too slow to process large volumes of data.
|
Package |
Description |
Reference |
How to Get It |
|
python |
A Python interpreter for best interactive experiences. Intel provides a robust and a secure build suitable for production enterprise environments. |
conda install python=[version] -c intel Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
|
numba |
A just-in-time (JIT) compiler for a subset of Python and NumPy. It can generate highly efficient native code leveraging the latest Intel instructions such as Intel® Advanced Vector Extensions 2 (Intel® AVX2) and Intel® Advanced Vector Extensions 512 (Intel® AVX-512). It is also capable of generating multithreaded code through OpenMP* or oneAPI Threading Building Blocks (oneTBB) that uses all available CPU cores |
conda install numba -c intel Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
|
cython |
An optimizing static compiler for Python, which is useful in writing C extensions for Python. Unlike Numba, which supports a subset of Python, Cython is a superset of the Python language.
|
conda install cython -c intel Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
Advanced Programming Packages
These are essential packages that enable fine-grained controls for data management, device management, concurrency, and parallelism.
|
Package |
Description |
Reference |
How to Get It |
|
tbb4py |
A library of composable multithreading based on oneTBB. It enables composability of two or more multithreaded modules by using the oneTBB library as a common work scheduler. oneTBB is a flexible performance library that supports scalable parallel programming using C++ code and is compliant with the ISO C++ standard. |
conda install tbb4py -c intel pip install tbb4py Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud
|
|
|
smp |
A library that controls thread affinity and static partitioning. Useful in static multiprocessing (SMP) programs where each process uses multithreaded library. Lack of affinity or partitioning controls provided by SMP leads to quadratic oversubscription of CPU cores by significantly degrading performance. |
conda install smp -c intel pip install smp Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
|
mpi4py |
A Python binding for MPI. Comes with Intel® MPI Library acceleration. |
conda install mpi4py -c intel python -m pip install mpi4py Download Intel Distribution for Python Access Intel Developer Cloud |
|
|
dpctl |
A Data Parallel Control library that helps to manage CPU and GPU devices through Python. It also implements the Python Array API standard. Learn more about Data Parallel Extensions for Python. |
conda install dpctl -c intel Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
Development Packages and Runtimes
Runtime packages are foundational for enabling Python performance. However, if you're a developer of a Python project that relies on built-in Intel libraries, you will also need the related development and runtime packages to ship your project.
|
Package |
Description |
Reference |
How to Get It |
|
mkl |
A Python package that provides all you need to ship oneMKL with your project. |
conda install mkl -c intel pip install mkl Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
|
mkl-devel |
A development package for oneMKL. |
conda install mkl-devel -c intel pip install mkl-devel |
|
|
mkl-dpcpp |
A Python package that provides all you need to ship oneMKL with your project. Learn more about Data Parallel Extensions for Python. |
conda install mkl-dpcpp -c intel pip install mkl-dpcpp Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
|
mkl-devel-dpcpp |
A development package that provides all you need for oneMKL development for SYCL devices. Learn more about Data Parallel Extensions for Python. |
conda install mkl-devel-dpcpp -c intel pip install mkl-devel-dpcpp |
|
|
daal |
A Python package that provides all you need to ship oneDAL with your project. |
conda install dal -c intel pip install dal Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
|
dal-devel |
A development package that provides all you need for development with oneDAL. |
conda install -c intel dal-devel pip install dal-devel |
|
|
ipp |
Python package that provides all you need to ship Intel® Integrated Performance Primitives (Intel® IPP) with your project. |
conda install ipp -c intel pip install ipp |
|
|
ipp-devel |
A development package that provides all you need for development with Intel IPP. |
conda install ipp-devel -c intel pip install ipp-devel |
|
|
tbb |
A Python package that provides all you need to ship oneTBB with your project. |
conda install tbb -c intel pip install tbb Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
|
tbb-devel |
A development package that provides all you need for development with oneTBB. |
conda install tbb-devel -c intel pip install tbb-devel |
|
|
impi-rt |
Python package that provides all you need to ship Intel® MPI with your project. |
conda install -c intel impi-rt pip install impi-rt Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
|
intel-openmp |
A Python package that provides all you need to ship OpenMP* with your project. |
conda install intel-openmp -c intel pip install intel-openmp Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
|
intel-opencl-rt |
A Python package that provides all you need to ship OpenCL™ Runtimes with your project. |
conda install intel-opencl-rt -c intel pip install intel-opencl-rt Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
|
icc-rt |
A Python package that provides all you need to ship your project that has native code compiled with Intel® C and Intel® C++ compilers. |
conda install icc_rt -c intel pip install icc-rt Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
|
fortran-rt |
A Python package that provides all you need to ship your project that has native code compiled with Intel® Fortran Compiler. |
conda install fortran_rt -c intel pip install fortran-rt Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
|
|
dpcpp-cpp-rt |
A Python package that provides all you need to ship your project and that has native code compiled with Intel® oneAPI DPC++ Compiler. Learn more about Data Parallel Extensions for Python.
|
Intel oneAPI DPC++/C++ Compiler Developer Guide and Reference |
conda install dpcpp-cpp-rt -c intel pip install dpcpp-cpp-rt Download Intel Distribution for Python Download Intel AI Analytics Toolkit Access Intel Developer Cloud |
Additional Recommended Packages
Intel Distribution for Python is an extensible platform that allows you to equip your numerical and data science workflow with all necessary tools and libraries. By using conda and PIP package managers, you can install additional packages not included in the stand-alone version.
Visualization and Interactive Environments
Python is ideal for interactive development. There are many Python tools and libraries that make interactive development easy and versatile. Following are a few essential tools and libraries:
|
Package |
Description |
|
A library for interactive visualizations inside browsers. |
|
|
A web-based interactive development environment for Jupyter* Notebooks, code, and data. It supports a wide range of workflows in data science, scientific computing, and machine learning. |
|
|
Plotting library with object-oriented API for embedding plots into applications. |
|
|
Web-based library for out-of-the-box interactive data visualizations supporting multiple linked views, animation, and crosstalk integration. |
|
|
Library for visualization of oriented and nonoriented graphs. |
|
|
Based on Matplotlib and serves a useful tool for visualizing statistical models, such as heatmaps and other visualizations that summarize data. |
Data Processing and Modeling
Data processing is an essential part of numerical and data science workflows. It includes data collection, ingestion, preprocessing, normalization, transformation, aggregation, and analysis. Data processing is based on numerical and statistical algorithms. Data analysis serves a basis for modeling of phenomena.
|
Package |
Description |
|
A foundational library for operations on DataFrames. |
|
|
Accelerates pandas workflows and scales DataFrame operations across multiple nodes. Intel® Distribution of Modin* integrates with OmniSci* for best performance. |
|
|
A library for symbolic mathematics. |
|
|
A library of essential statistical models, statistical tests, and statistical data exploration. |
|
|
Python interfaces to Apache* Arrow, columnar data format optimized for in-memory DataFrame operations, query processing, and I/O (CSV, Parquet, and so on). |
Image Processing and Computer Vision
Many scientific and AI applications work with images. Following are a few essential tools and libraries for image processing and computer vision.
|
Package |
Description |
|
Comprehensive library of image processing routines. The library is an image processing toolbox for SciPy. |
|
|
Library targeting real-time computer vision problems. |
Deep Learning
Deep learning and machine learning are special models, which allow a machine to automatically learn from a data without programming it explicitly.
|
Package |
Description |
|
TensorFlow* provides tutorials, examples, and other resources to speed up model building and create scalable machine learning solutions |
|
|
An open source machine learning framework that accelerates the path from research prototyping to production deployment |
|
|
Keras is an open source neural network library designed to provide fast experimentation with deep neural networks |
Natural Language Processing
Find useful Python tools for processing and analyzing human language data.
|
Package |
Description |
|
A toolkit for natural language processing, which includes libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning. |
|
|
A library for unsupervised topic modeling and natural language processing that uses advanced statistical machine learning. |
Data Crawling
Data scraping is a method of ingesting big data from a web. Data is typically extracted either by using direct APIs or via a general-purpose web crawler.
|
Package |
Description |
|
An open source framework for extracting the data you need from websites. |
|
|
A library that makes it easy to scrape information from web pages. |
Data Management, Device Management, Concurrency, Parallelism & Communication
These are essential packages that enable fine-grained controls.
|
Package |
Description |
|
A Python library for async concurrency and parallel I/O. |
|
|
A library for concurrency based on coroutines. It provides many useful abstractions for tasks, sockets, files, locks, and queues. |
|
|
A library for HTTP programming in a pythonic way. |
Package and Environment Managers
Python package and environment managers are essential tools for productivity. These tools allow you to install additional, update existing, and remove no longer needed packages from a Python environment. They also manage multiple Python environments customized by a user for multiple purposes or needs.
|
Package |
Description |
|
An advanced implementation of conda optimized for performance. It allows parallel downloading of repository data and package files as well as much faster dependency resolution. |