鱼皮AI导航
鱼皮AI导航
  • 主页
  • 社区
  • AI工具
  • AI知识库
  • AI提示词
  • MCP
  • Skills
  • AI学习资源
  • AI资讯
  • GitHub开源
登录/注册

bloblang-authoring

支持自然语言描述到可执行 Bloblang 脚本的端到端生成与验证,覆盖数据结构转换、字段映射、多格式解析、时间处理、条件过滤及错误恢复等全流程,确保每次输出均为通过真实数据测试的可靠脚本。

数据与分析内容与媒体
访问 GitHub
文档Skills.MD
name
bloblang-authoring
description
This skill should be used when users need to create or debug Bloblang transformation scripts. Trigger when users ask about transforming data, mapping fields, parsing JSON/CSV/XML, converting timestamps, filtering arrays, or mention "bloblang", "blobl", "mapping processor", or describe any data transformation need like "convert this to that" or "transform my JSON".

Redpanda Connect Bloblang Script Generator

Create working, tested Bloblang transformation scripts from natural language descriptions.

Objective

Generate a Bloblang (blobl) script that correctly transforms the user's input data according to their requirements. The script MUST be tested before presenting it.

Setup

This skill requires rpk rpk connect, python3, and jq. See the SETUP for installation instructions.

Tools

Script format-bloblang.sh

Generates category-organized Bloblang reference files in XML format. Run once at the start of each session before searching for functions/methods.

▼
bash
复制代码
# Usage: ./resources/scripts/format-bloblang.sh
  • No arguments
  • Generates category files organized by type (e.g., functions-General.xml, methods-String_Manipulation.xml)
  • Outputs generated files to a versioned directory
  • Outputs the directory path to stdout (capture in BLOBLREF_DIR variable for later use)
  • Each XML file contains structured function/method definitions with parameters, descriptions, and examples

Functions

Generated function files have functions-<Category>.xml names and contain functions relevant to that category.

  • functions-Encoding.xml - Schema registry headers
  • functions-Environment.xml - Environment vars, files, timestamps, hostname
  • functions-Fake_Data_Generation.xml - Fake data generation
  • functions-General.xml - Bytes, counter, deleted, ksuid, nanoid, uuid, random, range, snowflake
  • functions-Message_Info.xml - Batch index, content, error, metadata, span links, tracing IDs
  • etc.

The function XML tag format:

  • name attribute - function name
  • params attribute - comma-separated list of parameters with types, format <name>:<type> or empty string if no parameters
  • body - description of function purpose and usage
  • example XML subtag
    • summary attribute (optional) - brief description of the example
    • body - code block demonstrating usage

Example function definition:

▼
xml
复制代码
<function name="random_int" params="seed:query expression, min:integer, max:integer"> Generates a pseudo-random non-negative 64-bit integer. Use this for creating random IDs, sampling data, or generating test values. Provide a seed for reproducible randomness, or use a dynamic seed like `timestamp_unix_nano()` for unique values per mapping instance. Optional `min` and `max` parameters constrain the output range (both inclusive). For dynamic ranges based on message data, use the modulo operator instead: `random_int() % dynamic_max + dynamic_min`. <example> root.first = random_int() root.second = random_int(1) root.third = random_int(max:20) root.fourth = random_int(min:10, max:20) root.fifth = random_int(timestamp_unix_nano(), 5, 20) root.sixth = random_int(seed:timestamp_unix_nano(), max:20) </example> <example summary="Use a dynamic seed for unique random values per mapping instance."> root.random_id = random_int(timestamp_unix_nano()) root.sample_percent = random_int(seed: timestamp_unix_nano(), min: 0, max: 100) </example> </function>

Methods

Generated method files have methods-<Category>.xml names and contain methods relevant to that category.

  • methods-Encoding_and_Encryption.xml - Base64, compression, hashing, encryption
  • methods-General.xml - Basic operations, type checking
  • methods-GeoIP.xml - GeoIP lookups
  • methods-JSON_Web_Tokens.xml - JWT operations
  • methods-Number_Manipulation.xml - Arithmetic, rounding, formatting
  • methods-Object___Array_Manipulation.xml - Filtering, mapping, sorting, merging
  • methods-Parsing.xml - JSON, CSV, XML, protocol buffer parsing
  • methods-Regular_Expressions.xml - Regex matching and replacement
  • methods-SQL.xml - SQL operations
  • methods-String_Manipulation.xml - Case, trimming, splitting, formatting
  • methods-Timestamp_Manipulation.xml - Parsing, formatting, timezone conversion
  • methods-Type_Coercion.xml - Type conversions
  • etc.

The method XML tag format:

  • name attribute - function name
  • params attribute - comma-separated list of parameters with types, format <name>:<type> or empty string if no parameters
  • body - description of function purpose and usage
  • example XML subtag
    • summary attribute (optional) - brief description of the example
    • body - code block demonstrating usage

Example method definition:

▼
xml
复制代码
<method name="ts_format" params="format:string, tz:string"> Formats a timestamp into a string using the specified format layout. <example> root.formatted = this.timestamp.ts_format("2006-01-02T15:04:05Z07:00") </example> </method>

Grep Search

Lists Available functions and methods without loading full files.

▼
bash
复制代码
# List all available functions and methods by name grep -hE '<(function|method) name=' "$BLOBLREF_DIR" # Search by keyword (searches names, descriptions, params, examples) grep -i "timestamp" "$BLOBLREF_DIR" # Search by parameter name (e.g., find all with "format" parameter) grep 'params="[^"]*format' "$BLOBLREF_DIR"
  • Requires BLOBLREF_DIR set to the directory output by format-bloblang.sh

Script test-blobl.sh

Tests a Bloblang script against input data. Executes the transformation and returns results or errors. Can be run repeatedly during iteration.

▼
bash
复制代码
# Usage: ./resources/scripts/test-blobl.sh <target-directory>
  • Requires data.json (input) and script.blobl (transformation) in the target directory
  • Returns transformed data or error messages

Bloblang

Bloblang (blobl) is Redpanda Connect's native mapping language for transforming message data. It's designed for readability and safely reshaping documents of any structure.

Core Concepts

Assignment: Create new documents by assigning values to paths.

  • root = the new document being created
  • this = the input document being read
▼
bloblang
复制代码
# Copy entire input root = this # Create specific fields root.id = this.thing.id root.type = "processed" # In: {"thing":{"id":"abc123"}} # Out: {"id":"abc123","type":"processed"}

Field Paths: Use dot notation for nested fields. Use quotes for special characters:

▼
bloblang
复制代码
root.user.name = this.customer.full_name root."foo.bar".baz = this."field with spaces"

Literals: Numbers, booleans, strings, null, arrays, and objects:

▼
bloblang
复制代码
root = { "count": 42, "active": true, "items": ["a", "b", "c"], "nested": {"key": "value"} }

Functions and Methods

Functions generate values (no target needed):

▼
bloblang
复制代码
root.id = uuid_v4() root.timestamp = now() root.hostname = hostname()

Methods transform values (called on a target with .):

▼
bloblang
复制代码
root.upper = this.name.uppercase() root.formatted = this.date.ts_parse("2006-01-02").ts_format("Mon Jan 2") root.sorted = this.items.sort()

Methods can be chained:

▼
bloblang
复制代码
root.clean = this.text.trim().lowercase().replace_all("_", "-")

Methods require a target (called with .), while functions do not. Check the XML reference files to determine correct usage:

▼
bloblang
复制代码
# Bad: floor() is a method, not a function root.rounded = floor(this.value) # Error: floor is not a function # Good: Call floor() as a method on a value root.rounded = this.value.floor() # Bad: uuid_v4() is a function, not a method root.id = this.uuid_v4() # Error: uuid_v4 is not a method # Good: Call uuid_v4() as a function root.id = uuid_v4()

Discovering Available Functions & Methods

Bloblang provides hundreds of functions and methods organized into categories. Start with these foundational categories that cover common use cases:

  • functions-General.xml - Core utility functions (uuid_v4, timestamp, random, etc.)
  • functions-Message_Info.xml - Message metadata access (hostname, env, content_type, etc.)
  • methods-General.xml - Universal transformations (type conversions, existence checks, etc.)

For specialized needs, consult domain-specific categories: strings (uppercase, trim, regexp), timestamps (ts_parse, ts_format), arrays (map_each, filter), objects (keys, values), encoding (base64, json), and more.

Discovery tools:

  • Run format-bloblang.sh to generate category-organized XML reference files in a versioned directory
  • Use grep patterns to search function/method names, descriptions, parameters, and examples across categories
  • Read specific category XML files for structured definitions with complete function signatures, parameter details, and usage examples

Control Flow

Conditionals (if/else):

▼
bloblang
复制代码
root.category = if this.score >= 80 { "high" } else if this.score >= 50 { "medium" } else { "low" }

Pattern Matching (match):

▼
bloblang
复制代码
root.sound = match this.animal { "cat" => "meow" "dog" => "woof" "cow" => "moo" _ => "unknown" # Catch-all }

Coalescing (try multiple paths with |):

▼
bloblang
复制代码
# Use first non-null value from alternative fields root.content = this.article.body | this.comment.text | "no content" # Try different nested paths root.id = this.data.(primary_id | secondary_id | backup_id)

Note: Use | for alternative field paths (missing fields), use .catch() for operation failures (parse errors, type mismatches).

Common Operations

Deletion:

▼
bloblang
复制代码
root = this root.password = deleted() # Remove field # Or filter entire message root = if this.spam { deleted() }

Variables (reuse values without adding to output):

▼
bloblang
复制代码
let user_id = this.user.id let enriched = this.user.name + " (" + $user_id + ")" root.display_name = $enriched root.user_id = $user_id

IMPORTANT: Variables must be declared at the top level, not inside if, match, or other blocks.

▼
bloblang
复制代码
# Bad: Will cause "expected }" parse error root.age = if this.birthdate != null { let parsed = this.birthdate.ts_parse("2006-01-02") # let not allowed here! $parsed.ts_unix() } # Good: Declare variables at top level let parsed = this.birthdate.ts_parse("2006-01-02").catch(null) root.age = if $parsed != null { $parsed.ts_unix() } else { null }

Named mappings: (reusable scripts)

▼
bloblang
复制代码
map extract_user { root.id = this.user_id root.name = this.full_name root.email = this.contact.email } root.customer = this.customer_data.apply("extract_user") root.vendor = this.vendor_data.apply("extract_user")

Error Handling (provide fallback values):

▼
bloblang
复制代码
# Catch errors from any point in the chain root.count = this.items.length().catch(0) root.parsed = this.data.parse_json().catch({}) # Catch missing/null values root.name = this.user.name.or("anonymous") # Multi-format parsing with catch chains # Store value in variable for reliable access in catch fallbacks let date_str = this.date root.parsed = $date_str.ts_parse("2006-01-02").catch( $date_str.ts_parse("2006/01/02") ).catch(null)

IMPORTANT: When using .catch() with fallback expressions that reference this.field, store the field in a variable first. Context references in catch chains can be unreliable:

▼
bloblang
复制代码
# Risky: Context may not be preserved in catch root.parsed = this.date.ts_parse("2006-01-02").catch( this.date.ts_parse("2006/01/02") # this.date might not work here ) # Safe: Store in variable first let date_str = this.date root.parsed = $date_str.ts_parse("2006-01-02").catch( $date_str.ts_parse("2006/01/02") # variable reference is reliable )

Metadata:

▼
bloblang
复制代码
# Read metadata with @ or metadata() root.topic = @kafka_topic root.partition = @kafka_partition # Set metadata meta output_key = this.id meta content_type = "application/json"

Common Edge Case Patterns

Safe field access with fallbacks

▼
bloblang
复制代码
# Bad: Will fail if user or name is missing root.name = this.user.name # Good: Provides fallback chain root.name = this.user.name.or("anonymous") root.name = this.(user.name | profile.display_name | "unknown")

Safe collection operations

▼
bloblang
复制代码
# Bad: Will fail on empty array root.first = this.items[0] # Good: Handles empty arrays root.first = if this.items.length() > 0 { this.items[0] } else { null } root.first = this.items[0].catch(null)

Safe parsing with error recovery

▼
bloblang
复制代码
# Bad: Will fail on invalid JSON root.data = this.payload.parse_json() # Good: Provides fallback on parse failure root.data = this.payload.parse_json().catch({}) root.data = this.payload.parse_json().catch(this.payload) # Keep original on failure

Safe type coercion

▼
bloblang
复制代码
# Bad: Assumes field is already a string root.id = this.user_id.uppercase() # Good: Converts to string first root.id = this.user_id.string().uppercase() root.count = this.total.number().catch(0)

IMPORTANT: Arithmetic operations on null values fail silently. Always check for null or use .catch() to provide fallbacks:

▼
bloblang
复制代码
# Bad: Fails silently if price is null root.total = this.price * this.quantity # Good: Check for null before operations root.total = if this.price != null && this.quantity != null { this.price * this.quantity } else { null } # Also good: Use catch to handle null gracefully root.total = (this.price * this.quantity).catch(null)

Workflow

  1. Understand - Analyze input structure, desired output, and required transformations

    • Ambiguous requirements: If transformation goal is unclear, ask clarifying questions before proceeding (e.g., "Should missing fields be omitted or set to null?", "How should arrays with mixed types be handled?")
    • Missing sample data: If user doesn't provide input example, request it explicitly - never proceed with assumptions
    • Complex multistep transformations: Break down into logical phases (parse → transform → filter → format) and confirm approach with user
  2. Discover - Generate category files to versioned directory (capture BLOBLREF_DIR from script output), identify relevant categories, read specific category XML files to find actual Bloblang functions/methods (NEVER guess)

  3. Develop - Write valid Bloblang syntax using discovered functions (root for output, this for input, chain methods, handle nulls)

  4. Validate - Test script with sample input data, verify output matches expectations, iterate on errors until working

    • Test edge cases: Missing fields, null values, invalid formats, empty collections
    • Iterate: Fix syntax errors first (variable placement, method chains), then logic errors
  5. Deliver - Write the working script and example input to files (script.blobl, data.json), present the tested output, document any assumptions

Critical: Never present untested code. All scripts must be validated before showing to user.

数据与分析
内容与媒体

快捷安装

复制到终端去安装 Skill

npxskillsaddredpanda-data/connect--skill"bloblang-authoring"
站长:程序员鱼皮 老鱼简历 商务合作 增值电信业务经营许可证
2026 鱼皮AI导航 | 沪ICP备19026706号-6
站长:程序员鱼皮 老鱼简历 商务合作 增值电信业务经营许可证
2026 鱼皮AI导航 | 沪ICP备19026706号-6
🔥热门工具
朱雀AI检测朱雀AI检测
朱雀AI检测助手朱雀AI检测助手
LovartLovart
老鱼简历老鱼简历
WritePass官网WritePass官网
面多多面多多
CursorCursor
星月写作星月写作
朱雀AI检测助手朱雀AI检测助手
零感AI零感AI
Skills 推荐
definition-of-done-generator
自动生成项目完成标准的规范定义与配置文件,提供符合企业级工作流要求的最佳实践指导,输出可直接投入生产环境的代码和验证结果,适用于需要标准化流程审批、合规审查及跨系统集成的复杂项目管理场景。
sagemaker-endpoint-deployer
自动化部署机器学习模型至云端服务端点,支持从配置生成到生产验证的全流程,适用于模型上线、服务监控与推理优化等场景,确保部署方案符合工业级标准。
docx
支持专业级 Word 文档的全生命周期处理,包括从零创建结构化文档、精准编辑内容、管理修订痕迹与批注、提取纯文本或原始 XML 数据、转换为 PDF 或图像用于视觉分析,适用于法律、学术及企业场景下的高保真文档协作与自动化处理。
mermaid-sequence-diagram-creator
自动生成和优化序列图内容,支持可视化文档的快速构建与标准化输出,适用于图表设计、技术文档及演示材料的高效创作场景。
skills-search
提供对 Claude Code 技能生态的全周期管理能力,支持按功能关键词检索技能、查看详细信息、安装或卸载指定技能,并区分用户级与项目级作用域,覆盖发现、评估、部署及维护各环节。