Skip To Main Content
Intel logo - Return to the home page
My Tools

Select Your Language

  • Bahasa Indonesia
  • Deutsch
  • English
  • Español
  • Français
  • Português
  • Tiếng Việt
  • ไทย
  • 한국어
  • 日本語
  • 简体中文
  • 繁體中文
Sign In to access restricted content

Using Intel.com Search

You can easily search the entire Intel.com site in several ways.

  • Brand Name: Core i9
  • Document Number: 123456
  • Code Name: Emerald Rapids
  • Special Operators: “Ice Lake”, Ice AND Lake, Ice OR Lake, Ice*

Quick Links

You can also try the quick links below to see results for most popular searches.

  • Product Information
  • Support
  • Drivers & Software

Recent Searches

Sign In to access restricted content

Advanced Search

Only search in

Sign in to access restricted content.

The browser version you are using is not recommended for this site.
Please consider upgrading to the latest version of your browser by clicking one of the following links.

  • Safari
  • Chrome
  • Edge
  • Firefox

Python* Data Science at Scale: Speed Up Your End-to-End Workflow

Python* Data Science at Scale: Speed Up Your End-to-End Workflow

@IntelDevTools

Subscribe Now

Stay in the know on all things CODE. Updates are delivered to your inbox.

Sign Up

Overview

Data scientists and AI developers need the ability to explore and experiment with extremely large datasets as they converge on novel solutions for deployment in production applications. Exploration and experimentation means a lot of iteration, which is only feasible with fast turnaround times. While model training performance is an important part, the entire end-to-end process must be addressed. Loading, exploring, cleaning, and adding features to large datasets can often be so time-consuming that it limits exploration and experimentation. And responsiveness during inference is often crucial once a model is deployed.

Many of the solutions for large-scale AI development require installing new packages and rewriting code to use their APIs. For instance, data scientists and AI developers often use pandas to load data for machine learning applications. But once the size of the dataset gets to about 100 MB or larger, loading and cleaning the data really slows down because pandas is single-core only.

As a result, developers must change their workflow to use different data loading and preprocessing, such as switching to Apache Spark*, which requires data scientists to learn the Spark API and overhaul their code to integrate it. This is usually an inopportune time to make such changes and is not a good use of data scientists’ and AI developers’ skills.

Intel has been working to improve performance of popular Python* libraries while maintaining the usability of Python, by implementing the key underlying algorithms in built-in code using oneAPI performance libraries. This delivers concurrency at multiple levels, such as vectorization, multithreading, and multiprocessing with minimal impact on existing code. For example:

  • Modin* scales pandas DataFrames to multiple cores with a single line of code change.
  • Intel® Optimization for PyTorch* or Intel® Optimization for TensorFlow* accelerate deep learning training and inference.
  • Intel® Extension for Scikit-learn* or XGBoost optimized for Intel architecture speed up machine learning algorithms with no code changes.

In this session, see how to accelerate your end-to-end workflow with these technologies via a demonstration using the full New York City taxi fare dataset.

 

Presenters

  • Rachel Oberman, technical consulting engineer, Intel
  • Todd Tomashek, machine learning engineer, Intel
  • Albert DeFusco, principal data scientist, Anaconda*

 

Featured Software

Get these Intel-optimized versions of your Python libraries as part of the AI Tools, or download them as stand-alone components:

  • Modin
  • Intel Optimization for PyTorch
  • Intel Optimization for TensorFlow
  • Intel Extension for Scikit-learn
  • XGBoost Optimized for Intel Architecture

 

Additional Resources

AI Tools, Libraries, and Framework Optimizations

 

Jump to:

You May Also Like
 


 

You May Also Like

Related Articles & Blogs

Optimize End-to-End AI Pipelines

Speed up Databricks* Runtime for Machine Learning with Intel®-optimized Libraries

Scale Your pandas Workflow with Modin—No Rewrite Required

One-Line Code Changes Boost Data Analytics Performance

Related Webinars

Optimize Deep Learning Workloads Using PyTorch Optimized by Intel

Achieve Up to 1.77x Boost Ratio for Your PyTorch AI Workloads

Seamlessly Scale pandas Workloads with a Single Code-Line Change

Drive 2x Performance into Your scikit-learn Machine Learning Tasks

Related Podcast

An Open Road to Swift DataFrame Scaling

  • Company Overview
  • Contact Intel
  • Newsroom
  • Investors
  • Careers
  • Corporate Responsibility
  • Inclusion
  • Public Policy
  • © Intel Corporation
  • Terms of Use
  • *Trademarks
  • Cookies
  • Privacy
  • Supply Chain Transparency
  • Site Map
  • Recycling
  • Your Privacy Choices California Consumer Privacy Act (CCPA) Opt-Out Icon
  • Notice at Collection

Intel technologies may require enabled hardware, software or service activation. // No product or component can be absolutely secure. // Your costs and results may vary. // Performance varies by use, configuration, and other factors. Learn more at intel.com/performanceindex. // See our complete legal Notices and Disclaimers. // Intel is committed to respecting human rights and avoiding causing or contributing to adverse impacts on human rights. See Intel’s Global Human Rights Principles. Intel’s products and software are intended only to be used in applications that do not cause or contribute to adverse impacts on human rights.

Intel Footer Logo