Opensearch Vector Search Skill

OpenSearch Vector Search Expert

An OpenClaw AgentSkill for Amazon OpenSearch vector search (k-NN). Provides comprehensive guidance on configuration, cluster tuning, quantization, cost optimization, instance sizing, and live cluster analysis.

Features

• Vector Search Configuration — FAISS/HNSW parameter tuning, disk mode, space type selection
• Capacity Planning — HNSW memory formula calculations, instance sizing recommendations
• Quantization Techniques — FP16, Byte, Binary, Product Quantization with recall/cost tradeoffs
• Cost Estimation — Real-time AWS pricing via Pricing API, monthly cost projections
• Cluster Tuning — JVM, thread pools, shard strategies, node roles
• Performance Benchmarks — QPS/latency/recall data for various configurations
• Live Cluster Analyzer 🆕 — Connect to any OpenSearch cluster, auto-discover k-NN indices, analyze vector configs, and generate optimization recommendations (read-only)

Install

npx clawhub@latest install opensearch-vector-search

Or manually copy to your OpenClaw skills directory:

cp -r . ~/.openclaw/skills/opensearch-vector-search/

Project Structure

├── SKILL.md                              # Skill definition and workflows
├── references/
│   ├── vector-search.md                  # k-NN, HNSW, disk mode guide
│   ├── quantization-techniques.md        # Compression techniques comparison
│   ├── cost-optimization.md              # Instance sizing, memory formulas, cost cases
│   ├── cluster-tuning.md                 # JVM, thread pools, node configuration
│   ├── performance-benchmarks.md         # QPS/latency/recall benchmark data
│   ├── indexing-strategies.md            # Index mapping, shard, lifecycle
│   ├── query-optimization.md             # Query tuning, caching, pagination
│   └── optimized-instances.md            # OR1/OR2/OM2/OI2 instance guide
├── scripts/
│   ├── get_opensearch_pricing.py         # AWS Pricing API query tool
│   └── analyze_cluster.py               # Live cluster analyzer (read-only)

Live Cluster Analyzer

Connect to an OpenSearch cluster and analyze vector search configurations:

# Full analysis
python3 scripts/analyze_cluster.py \
  --url https://my-cluster.us-east-1.es.amazonaws.com \
  -u admin -p MyPassword \
  --action all -f pretty

# Cluster overview only
python3 scripts/analyze_cluster.py \
  --url https://my-cluster:9200 \
  -u admin -p MyPassword \
  --action cluster-overview

# Specific index deep dive
python3 scripts/analyze_cluster.py \
  --url https://my-cluster:9200 \
  -u admin -p MyPassword \
  --action index-detail --index my_vectors

What it analyzes:

• Cluster health, node resources (memory/CPU/JVM), OpenSearch version
• Auto-discovers all k-NN enabled indices
• Vector field configs: engine, dimensions, HNSW params, quantization, disk mode
• Memory estimates using AWS official HNSW formula
• Shard distribution across nodes
• Auto-generated optimization recommendations with severity levels

Safety: This script is strictly read-only. It never creates, modifies, or deletes any indices or data.

Requirements

pip install opensearch-py

Cost Estimation

Query real-time AWS OpenSearch pricing:

# All instance prices in a region
python3 scripts/get_opensearch_pricing.py --region us-east-1

# Specific instance type
python3 scripts/get_opensearch_pricing.py --region us-east-1 --instance-type r7g.12xlarge --format json

Requires boto3 and valid AWS credentials.

Key Formulas

HNSW Memory (AWS Official)

Unquantized:  Memory = 1.1 × (4 × d + 8 × m) × num_vectors × (replicas + 1)
FP16 (2x):    Memory = 1.1 × (2 × d + 8 × m) × num_vectors × (replicas + 1)
Byte (4x):    Memory = 1.1 × (1 × d + 8 × m) × num_vectors × (replicas + 1)

Node Memory Allocation

JVM Heap = min(node_memory × 50%, 32GB)
KNN available = (node_memory - JVM Heap) × 75%

License

MIT-0 — Free to use, modify, and redistribute. No attribution required.