鱼皮AI导航
鱼皮AI导航
  • 主页
  • 社区
  • AI工具
  • AI知识库
  • AI提示词
  • MCP
  • Skills
  • AI学习资源
  • AI资讯
  • GitHub开源
登录/注册

polars

提供高性能数据处理能力,支持大规模数据集的快速读写、列式计算、条件筛选、分组聚合、窗口函数及多表连接,通过表达式API和惰性求值实现并行优化与查询计划自动优化。

数据与分析
访问 GitHub
文档Skills.MD
name
polars
description
Fast DataFrame library (Apache Arrow). Select, filter, group_by, joins, lazy evaluation, CSV/Parquet I/O, expression API, for high-performance data analysis workflows.

Polars

Overview

Polars is a lightning-fast DataFrame library for Python and Rust built on Apache Arrow. Work with Polars' expression-based API, lazy evaluation framework, and high-performance data manipulation capabilities for efficient data processing, pandas migration, and data pipeline optimization.

Quick Start

Installation and Basic Usage

Install Polars:

▼
python
复制代码
uv pip install polars

Basic DataFrame creation and operations:

▼
python
复制代码
import polars as pl # Create DataFrame df = pl.DataFrame({ "name": ["Alice", "Bob", "Charlie"], "age": [25, 30, 35], "city": ["NY", "LA", "SF"] }) # Select columns df.select("name", "age") # Filter rows df.filter(pl.col("age") > 25) # Add computed columns df.with_columns( age_plus_10=pl.col("age") + 10 )

Core Concepts

Expressions

Expressions are the fundamental building blocks of Polars operations. They describe transformations on data and can be composed, reused, and optimized.

Key principles:

  • Use pl.col("column_name") to reference columns
  • Chain methods to build complex transformations
  • Expressions are lazy and only execute within contexts (select, with_columns, filter, group_by)

Example:

▼
python
复制代码
# Expression-based computation df.select( pl.col("name"), (pl.col("age") * 12).alias("age_in_months") )

Lazy vs Eager Evaluation

Eager (DataFrame): Operations execute immediately

▼
python
复制代码
df = pl.read_csv("file.csv") # Reads immediately result = df.filter(pl.col("age") > 25) # Executes immediately

Lazy (LazyFrame): Operations build a query plan, optimized before execution

▼
python
复制代码
lf = pl.scan_csv("file.csv") # Doesn't read yet result = lf.filter(pl.col("age") > 25).select("name", "age") df = result.collect() # Now executes optimized query

When to use lazy:

  • Working with large datasets
  • Complex query pipelines
  • When only some columns/rows are needed
  • Performance is critical

Benefits of lazy evaluation:

  • Automatic query optimization
  • Predicate pushdown
  • Projection pushdown
  • Parallel execution

For detailed concepts, load references/core_concepts.md.

Common Operations

Select

Select and manipulate columns:

▼
python
复制代码
# Select specific columns df.select("name", "age") # Select with expressions df.select( pl.col("name"), (pl.col("age") * 2).alias("double_age") ) # Select all columns matching a pattern df.select(pl.col("^.*_id$"))

Filter

Filter rows by conditions:

▼
python
复制代码
# Single condition df.filter(pl.col("age") > 25) # Multiple conditions (cleaner than using &) df.filter( pl.col("age") > 25, pl.col("city") == "NY" ) # Complex conditions df.filter( (pl.col("age") > 25) | (pl.col("city") == "LA") )

With Columns

Add or modify columns while preserving existing ones:

▼
python
复制代码
# Add new columns df.with_columns( age_plus_10=pl.col("age") + 10, name_upper=pl.col("name").str.to_uppercase() ) # Parallel computation (all columns computed in parallel) df.with_columns( pl.col("value") * 10, pl.col("value") * 100, )

Group By and Aggregations

Group data and compute aggregations:

▼
python
复制代码
# Basic grouping df.group_by("city").agg( pl.col("age").mean().alias("avg_age"), pl.len().alias("count") ) # Multiple group keys df.group_by("city", "department").agg( pl.col("salary").sum() ) # Conditional aggregations df.group_by("city").agg( (pl.col("age") > 30).sum().alias("over_30") )

For detailed operation patterns, load references/operations.md.

Aggregations and Window Functions

Aggregation Functions

Common aggregations within group_by context:

  • pl.len() - count rows
  • pl.col("x").sum() - sum values
  • pl.col("x").mean() - average
  • pl.col("x").min() / pl.col("x").max() - extremes
  • pl.first() / pl.last() - first/last values

Window Functions with over()

Apply aggregations while preserving row count:

▼
python
复制代码
# Add group statistics to each row df.with_columns( avg_age_by_city=pl.col("age").mean().over("city"), rank_in_city=pl.col("salary").rank().over("city") ) # Multiple grouping columns df.with_columns( group_avg=pl.col("value").mean().over("category", "region") )

Mapping strategies:

  • group_to_rows (default): Preserves original row order
  • explode: Faster but groups rows together
  • join: Creates list columns

Data I/O

Supported Formats

Polars supports reading and writing:

  • CSV, Parquet, JSON, Excel
  • Databases (via connectors)
  • Cloud storage (S3, Azure, GCS)
  • Google BigQuery
  • Multiple/partitioned files

Common I/O Operations

CSV:

▼
python
复制代码
# Eager df = pl.read_csv("file.csv") df.write_csv("output.csv") # Lazy (preferred for large files) lf = pl.scan_csv("file.csv") result = lf.filter(...).select(...).collect()

Parquet (recommended for performance):

▼
python
复制代码
df = pl.read_parquet("file.parquet") df.write_parquet("output.parquet")

JSON:

▼
python
复制代码
df = pl.read_json("file.json") df.write_json("output.json")

For comprehensive I/O documentation, load references/io_guide.md.

Transformations

Joins

Combine DataFrames:

▼
python
复制代码
# Inner join df1.join(df2, on="id", how="inner") # Left join df1.join(df2, on="id", how="left") # Join on different column names df1.join(df2, left_on="user_id", right_on="id")

Concatenation

Stack DataFrames:

▼
python
复制代码
# Vertical (stack rows) pl.concat([df1, df2], how="vertical") # Horizontal (add columns) pl.concat([df1, df2], how="horizontal") # Diagonal (union with different schemas) pl.concat([df1, df2], how="diagonal")

Pivot and Unpivot

Reshape data:

▼
python
复制代码
# Pivot (wide format) df.pivot(values="sales", index="date", columns="product") # Unpivot (long format) df.unpivot(index="id", on=["col1", "col2"])

For detailed transformation examples, load references/transformations.md.

Pandas Migration

Polars offers significant performance improvements over pandas with a cleaner API. Key differences:

Conceptual Differences

  • No index: Polars uses integer positions only
  • Strict typing: No silent type conversions
  • Lazy evaluation: Available via LazyFrame
  • Parallel by default: Operations parallelized automatically

Common Operation Mappings

OperationPandasPolars
Select columndf["col"]df.select("col")
Filterdf[df["col"] > 10]df.filter(pl.col("col") > 10)
Add columndf.assign(x=...)df.with_columns(x=...)
Group bydf.groupby("col").agg(...)df.group_by("col").agg(...)
Windowdf.groupby("col").transform(...)df.with_columns(...).over("col")

Key Syntax Patterns

Pandas sequential (slow):

▼
python
复制代码
df.assign( col_a=lambda df_: df_.value * 10, col_b=lambda df_: df_.value * 100 )

Polars parallel (fast):

▼
python
复制代码
df.with_columns( col_a=pl.col("value") * 10, col_b=pl.col("value") * 100, )

For comprehensive migration guide, load references/pandas_migration.md.

Best Practices

Performance Optimization

  1. Use lazy evaluation for large datasets:

    ▼
    python
    复制代码
    lf = pl.scan_csv("large.csv") # Don't use read_csv result = lf.filter(...).select(...).collect()
  2. Avoid Python functions in hot paths:

    • Stay within expression API for parallelization
    • Use .map_elements() only when necessary
    • Prefer native Polars operations
  3. Use streaming for very large data:

    ▼
    python
    复制代码
    lf.collect(streaming=True)
  4. Select only needed columns early:

    ▼
    python
    复制代码
    # Good: Select columns early lf.select("col1", "col2").filter(...) # Bad: Filter on all columns first lf.filter(...).select("col1", "col2")
  5. Use appropriate data types:

    • Categorical for low-cardinality strings
    • Appropriate integer sizes (i32 vs i64)
    • Date types for temporal data

Expression Patterns

Conditional operations:

▼
python
复制代码
pl.when(condition).then(value).otherwise(other_value)

Column operations across multiple columns:

▼
python
复制代码
df.select(pl.col("^.*_value$") * 2) # Regex pattern

Null handling:

▼
python
复制代码
pl.col("x").fill_null(0) pl.col("x").is_null() pl.col("x").drop_nulls()

For additional best practices and patterns, load references/best_practices.md.

Resources

This skill includes comprehensive reference documentation:

references/

  • core_concepts.md - Detailed explanations of expressions, lazy evaluation, and type system
  • operations.md - Comprehensive guide to all common operations with examples
  • pandas_migration.md - Complete migration guide from pandas to Polars
  • io_guide.md - Data I/O operations for all supported formats
  • transformations.md - Joins, concatenation, pivots, and reshaping operations
  • best_practices.md - Performance optimization tips and common patterns

Load these references as needed when users require detailed information about specific topics.

数据与分析

快捷安装

复制到终端去安装 Skill

npxskillsadddavila7/claude-code-templates--skill"polars"
站长:程序员鱼皮 老鱼简历 商务合作 增值电信业务经营许可证
2026 鱼皮AI导航 | 沪ICP备19026706号-6
站长:程序员鱼皮 老鱼简历 商务合作 增值电信业务经营许可证
2026 鱼皮AI导航 | 沪ICP备19026706号-6
🔥热门工具
朱雀AI检测朱雀AI检测
朱雀AI检测助手朱雀AI检测助手
LovartLovart
老鱼简历老鱼简历
WritePass官网WritePass官网
面多多面多多
CursorCursor
星月写作星月写作
朱雀AI检测助手朱雀AI检测助手
零感AI零感AI
Skills 推荐
definition-of-done-generator
自动生成项目完成标准的规范定义与配置文件,提供符合企业级工作流要求的最佳实践指导,输出可直接投入生产环境的代码和验证结果,适用于需要标准化流程审批、合规审查及跨系统集成的复杂项目管理场景。
sagemaker-endpoint-deployer
自动化部署机器学习模型至云端服务端点,支持从配置生成到生产验证的全流程,适用于模型上线、服务监控与推理优化等场景,确保部署方案符合工业级标准。
docx
支持专业级 Word 文档的全生命周期处理,包括从零创建结构化文档、精准编辑内容、管理修订痕迹与批注、提取纯文本或原始 XML 数据、转换为 PDF 或图像用于视觉分析,适用于法律、学术及企业场景下的高保真文档协作与自动化处理。
skills-search
提供对 Claude Code 技能生态的全周期管理能力,支持按功能关键词检索技能、查看详细信息、安装或卸载指定技能,并区分用户级与项目级作用域,覆盖发现、评估、部署及维护各环节。
moai-foundation-core
构建高质量、可扩展的AI开发工作流,通过领域驱动设计与模块化架构实现需求规范先行,结合自动化质量门禁、精细化上下文管理与渐进式知识呈现,在保障安全与可维护性的同时提升开发效率与协作清晰度。