OpenObserve API Usage Guide for Log Management

OpenObserve API Usage Guide for Audit Log Management

1. 概述

1.1 目标

本文档旨在详细介绍 OpenObserve 的 API 使用方法，帮助用户通过 API 实现日志管理功能，包括日志摄入、查询、模糊匹配（类似 SQL 的 LIKE）、stream 管理等。文档基于 test stream 的审计日志场景，解决之前遇到的模糊匹配问题。

1.2 OpenObserve API 简介

OpenObserve 是一个开源的 observability 平台，支持日志、指标和追踪数据。API 是与 OpenObserve 交互的主要方式，支持以下功能：

• 日志摄入：通过 API 批量摄入日志数据。
• 日志查询：支持 SQL-like 语法查询日志，支持模糊匹配。
• Stream 管理：创建、更新、删除 stream，以及管理 stream 设置（如 full_text_search_keys）。
• 用户和组织管理：管理组织、用户和权限。
• 告警和仪表盘：配置告警规则和创建仪表盘。

1.3 认证

OpenObserve API 使用 HTTP Basic Authentication，所有请求必须包含 Authorization 头。认证信息是用户 ID 和密码的 Base64 编码。

生成 Authorization 头

• 用户 ID：admin@admin.com
• 密码：admin
• Base64 编码：echo -n "admin@admin.com:admin" | base64 -> YWRtaW5AYWRtaW4uY29tOmFkbWlu

示例

-H "Authorization: Basic YWRtaW5AYWRtaW4uY29tOmFkbWlu"

2. API 端点和使用方法

以下是 OpenObserve 的主要 API 端点及其使用方法，基于 OpenObserve 文档。

2.1 日志摄入 API

端点

• POST /api/{organization}/{stream}/_json

功能

批量摄入日志数据到指定 stream。OpenObserve 会根据数据自动推断 schema。

请求

• Content-Type：application/json
• Body：JSON 数组，每个元素是一条日志记录。

示例

摄入 3 条审计日志到 test stream：

curl -X POST \-H "Authorization: Basic YWRtaW5AYWRtaW4uY29tOmFkbWlu" \-H "Content-Type: application/json" \-d '[{"user": "test_user_create","path": "/test/path","category": "File Management","event": "Create Document","date": "2025-04-28 17:01:00","comment": "Created a new document","ip": "192.168.1.1"},{"user": "admin_user","path": "/admin/path","category": "User Management","event": "Update User","date": "2025-04-28 17:02:00","comment": "Updated user profile","ip": "192.168.1.2"},{"user": "create_team_lead","path": "/team/path","category": "Team Management","event": "Create Team","date": "2025-04-28 17:03:00","comment": "Created a new team","ip": "192.168.1.3"}]' \http://localhost:5080/api/default/test/_json

响应

{"code": 200,"status": [{"name": "test","successful": 3,"failed": 0}]
}

注意事项

• 如果 test stream 不存在，OpenObserve 会自动创建。
• 确保字段类型一致（例如 date 应始终为字符串格式）。

2.2 日志查询 API

端点

• POST /api/{organization}/{stream}/_search

功能

使用 SQL-like 语法查询日志，支持模糊匹配、聚合等操作。

请求

• Content-Type：application/json
• Body：
  • query.sql：SQL 查询语句。
  • query.start_time 和 query.end_time：查询时间范围（微秒）。
  • from 和 size：分页参数。

示例 1：基本查询

查询 test stream 的最新 10 条记录：

curl -X POST \-H "Authorization: Basic YWRtaW5AYWRtaW4uY29tOmFkbWlu" \-H "Content-Type: application/json" \-d '{"query": {"sql": "SELECT * FROM test","start_time": 0,"end_time": 999999999999999,"from": 0,"size": 10}}' \http://localhost:5080/api/default/test/_search

响应

{"took": 10,"hits": {"total": 3,"hits": [{"user": "test_user_create","path": "/test/path","category": "File Management","event": "Create Document","date": "2025-04-28 17:01:00","comment": "Created a new document","ip": "192.168.1.1"},{"user": "admin_user","path": "/admin/path","category": "User Management","event": "Update User","date": "2025-04-28 17:02:00","comment": "Updated user profile","ip": "192.168.1.2"},{"user": "create_team_lead","path": "/team/path","category": "Team Management","event": "Create Team","date": "2025-04-28 17:03:00","comment": "Created a new team","ip": "192.168.1.3"}]}
}

示例 2：模糊匹配查询

使用 str_match 实现模糊匹配（类似 LIKE '%create%'），查找 user 字段包含 “create” 的记录：

curl -X POST \-H "Authorization: Basic YWRtaW5AYWRtaW4uY29tOmFkbWlu" \-H "Content-Type: application/json" \-d '{"query": {"sql": "SELECT * FROM test WHERE str_match(user, '\''create'\'')","start_time": 0,"end_time": 999999999999999,"from": 0,"size": 10}}' \http://localhost:5080/api/default/test/_search

响应

{"took": 10,"hits": {"total": 2,"hits": [{"user": "test_user_create","path": "/test/path","category": "File Management","event": "Create Document","date": "2025-04-28 17:01:00","comment": "Created a new document","ip": "192.168.1.1"},{"user": "create_team_lead","path": "/team/path","category": "Team Management","event": "Create Team","date": "2025-04-28 17:03:00","comment": "Created a new team","ip": "192.168.1.3"}]}
}

示例 3：全局全文搜索

使用 match_all 在所有字段中查找 “create”：

curl -X POST \-H "Authorization: Basic YWRtaW5AYWRmiW4uY29tOmFkbWlu" \-H "Content-Type: application/json" \-d '{"query": {"sql": "SELECT * FROM test WHERE match_all('\''create'\'')","start_time": 0,"end_time": 999999999999999,"from": 0,"size": 10}}' \http://localhost:5080/api/default/test/_search

响应

{"took": 10,"hits": {"total": 3,"hits": [{"user": "test_user_create","path": "/test/path","category": "File Management","event": "Create Document","date": "2025-04-28 17:01:00","comment": "Created a new document","ip": "192.168.1.1"},{"user": "admin_user","path": "/admin/path","category": "User Management","event": "Update User","date": "2025-04-28 17:02:00","comment": "Updated user profile","ip": "192.168.1.2"},{"user": "create_team_lead","path": "/team/path","category": "Team Management","event": "Create Team","date": "2025-04-28 17:03:00","comment": "Created a new team","ip": "192.168.1.3"}]}
}

注意事项

• str_match 不需要字段在 full_text_search_keys 中，但如果启用全文搜索，查询效率可能更高。
• match_all 依赖 full_text_search_keys，否则可能无法匹配。

2.3 Stream 管理 API

端点 1：列出所有 Streams

• GET /api/{organization}/streams

功能

列出组织中的所有 stream，包括 schema 和设置。

示例

curl -X GET \-H "Authorization: Basic YWRtaW5AYWRtaW4uY29tOmFkbWlu" \http://localhost:5080/api/default/streams

响应

[{"name": "test","storage_type": "disk","stream_type": "logs","stats": {"doc_num": 3,"storage_size": 0.0003},"schema": [{"name": "_timestamp", "type": "Int64"},{"name": "category", "type": "Utf8"},{"name": "comment", "type": "Utf8"},{"name": "date", "type": "Utf8"},{"name": "event", "type": "Utf8"},{"name": "ip", "type": "Utf8"},{"name": "path", "type": "Utf8"},{"name": "user", "type": "Utf8"}],"settings": {"full_text_search_keys": [],"data_retention": 0}}
]

端点 2：更新 Stream 设置

• PUT /api/{organization}/streams/{stream}/settings

功能

更新 stream 的设置，例如 full_text_search_keys。

示例

尝试启用 user 字段的全文搜索：

curl -X PUT \-H "Authorization: Basic YWRtaW5AYWRtaW4uY29tOmFkbWlu" \-H "Content-Type: application/json" \-d '{"full_text_search_keys": ["user"]}' \http://localhost:5080/api/default/streams/test/settings

响应

{"code": 200,"message": "Settings updated successfully for stream test"
}

注意事项

• 之前尝试更新 full_text_search_keys 失败（Json deserialize error），建议通过 UI 手动设置，或者联系 OpenObserve 社区解决。

端点 3：删除 Stream

• DELETE /api/{organization}/streams/{stream}

功能

删除指定 stream 及其数据。

示例

curl -X DELETE \-H "Authorization: Basic YWRtaW5AYWRtaW4uY29tOmFkbWlu" \http://localhost:5080/api/default/streams/test

响应

{"code": 200,"message": "Stream test deleted successfully"
}

2.4 告警 API

端点

• POST /api/{organization}/alerts

功能

创建告警规则，例如当 event 字段包含 “Error” 时触发告警。

示例

创建告警，当 event 包含 “Error” 时触发：

curl -X POST \-H "Authorization: Basic YWRtaW5AYWRtaW4uY29tOmFkbWlu" \-H "Content-Type: application/json" \-d '{"name": "error_alert","stream": "test","query": "SELECT * FROM test WHERE str_match(event, '\''Error'\'')","condition": "num > 0","threshold": 1,"frequency": "5m","destination": "email","email": "admin@example.com"}' \http://localhost:5080/api/default/alerts

响应

{"code": 200,"message": "Alert created successfully"
}

3. 总结

OpenObserve 的 API 提供了强大的日志管理功能，支持摄入、查询、模糊匹配和 stream 管理。通过 str_match 可以实现类似 LIKE 的模糊匹配，满足审计日志场景的需求。建议进一步优化 full_text_search_keys 设置，并关注 OpenObserve 的版本更新以获取更多功能支持（如 REGEXP 或 LIKE）。