The Problem: 3.3MB Every Hour Is Wasteful
---
title: "STIX Feed Optimization: 4 Parameters That Cut Bandwidth 90%"
slug: stix-feed-optimization-guide-bandwidth-reduction
date: 2025-12-21
author: Patrick Duggan
tags: [stix, threat-intelligence, api, optimization, bandwidth]
category: Security Tips
featured: false
---
Our STIX 2.1 feed hit 1,017 indicators at 3.3MB yesterday. That's rich context: full MITRE ATT&CK mapping, multi-source correlation, behavioral analysis. But for consumers with:
• Limited storage (embedded SIEM appliances)
• Metered bandwidth (cloud egress charges)
• Strict ingestion limits (5MB/day on free tiers)
• Hourly polling schedules (don't need the full corpus every time)
...downloading 3.3MB every hour is wasteful. We analyzed our consumer churn patterns and found 35% of churned consumers cited storage/ingestion limits as the primary reason for stopping.
So we fixed it.
The 4 Optimization Parameters
1. Delta Feed: `?since=<ISO8601>`
What it does: Only returns indicators modified after the specified timestamp.
Bandwidth savings: ~90% (typical hourly poll)
# Get indicators modified in the last hour
curl -s "https://analytics.dugganusa.com/api/v1/stix-feed?since=$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ)" | jq .
Python example:
```python
from datetime import datetime, timedelta
import requests
Use case: Hourly/daily polling schedules. Get only what's changed.
2. Minimal Mode: `?minimal=true`
What it does: Strips verbose context (descriptions, external references, kill chain phases). Returns core fields only: type, id, pattern, confidence, labels.
Size reduction: ~85%
# Full indicator: ~2KB each
curl -s "https://analytics.dugganusa.com/api/v1/stix-feed" | jq '.objects[0]' | wc -c
# 2147• `type`, `spec_version`, `id`
• `created`, `modified`, `valid_from`
• `pattern`, `pattern_type`
• `confidence`
• `labels` (first 3 only)
• `description` (often 500+ characters)
• `external_references` (URLs, MITRE mappings)
• `kill_chain_phases` (ATT&CK tactics)
• `x_dugganusa_threat_intel` (our custom enrichment)
• `x_dugganusa_discovery` (detection metadata)
Use case: Blocklist generation, SIEM ingestion where you just need the IOC pattern.
3. Pagination: `?limit=N&offset=N`
What it does: Returns a slice of the feed for controlled ingestion.
# First 100 indicators
curl -s "https://analytics.dugganusa.com/api/v1/stix-feed?limit=100&offset=0" | jq .Python pagination loop:
```python
import requests
def fetch_paginated(limit=100):
offset = 0
all_indicators = []
while True:
response = requests.get(
f'https://analytics.dugganusa.com/api/v1/stix-feed?limit={limit}&offset={offset}'
)
data = response.json()
objects = data.get('objects', [])
if not objects:
break
all_indicators.extend(objects)
offset += limit
print(f"Fetched {len(all_indicators)} indicators...")
return all_indicators
indicators = fetch_paginated(limit=100)
print(f"Total: {len(indicators)}")
```
Use case: Appliances with 1MB request limits, rate-limited ingestion pipelines.
4. Combined: Maximum Efficiency
The power move: Combine all parameters.
# Delta + Minimal + Paginated = Maximum efficiency
curl -s "https://analytics.dugganusa.com/api/v1/stix-feed?since=2025-12-21T00:00:00Z&minimal=true&limit=100" | jq .
Result: Instead of 3.3MB, you get ~15KB of exactly what you need.
Consumption Patterns by Vertical
Enterprise SOC (24/7 Operations)
Goal: Real-time threat awareness, minimal latency
Pattern: Delta feed every 15 minutes, full context
# Cron job: */15 * * * *
SINCE=$(date -u -v-15M +%Y-%m-%dT%H:%M:%SZ)
curl -s "https://analytics.dugganusa.com/api/v1/stix-feed?since=$SINCE" >> /var/log/threat-intel/dugganusa.json
Why: SOCs need the full context (MITRE mapping, behavioral analysis) for triage decisions. 15-minute delta keeps bandwidth reasonable (~50KB/poll) while maintaining near-real-time coverage.
SIEM Ingestion (Splunk, Sentinel, Chronicle)
Goal: Blocklist updates, correlation searches
Pattern: Minimal mode, daily full refresh + hourly deltas
python
# Daily full refresh (overnight, off-peak)
# 0 3 * * * /path/to/daily-refresh.pyimport requests
import json
print(f"Updated {len(ips)} indicators")
```
Why: SIEMs care about the IOC pattern, not the full narrative. Minimal mode cuts 85% of payload. Daily refresh catches stragglers.
Embedded Security Appliance (5MB/day limit)
Goal: Stay within ingestion limits
Pattern: Paginated minimal, high-confidence only
# Daily budget: 5MB
# Our full feed: 3.3MB
# Solution: Minimal + high confidence + paginationWhy: Appliances with hard limits need surgical precision. Pagination + minimal + confidence filtering gets you the highest-value intel within budget.
Cloud-Native Security (Wiz, Orca, Prisma)
Goal: Enrich cloud findings with threat context
Pattern: Delta feed with full context, triggered by alerts
Microsoft pulls this feed daily. AT&T pulls this feed daily. Starlink pulls this feed daily. Get the DugganUSA STIX feed — $9/mo →
python
# Triggered by Wiz alert webhook
# Fetch fresh threat intel when investigatingimport requests
from datetime import datetime, timedelta
def enrich_finding(finding_ip):
# Get last 24 hours of intel
since = (datetime.utcnow() - timedelta(hours=24)).strftime('%Y-%m-%dT%H:%M:%SZ')
response = requests.get(f'https://analytics.dugganusa.com/api/v1/stix-feed?since={since}')
data = response.json()
return {'match': False}
Why: Cloud security tools generate findings that need context. On-demand enrichment with delta feed keeps queries fast.
Firewall/IDS/IPS Blocklist
Goal: IP blocklist, no context needed
Pattern: Minimal mode, extract IPs only
# Generate IP blocklist for iptables/pf/nftables
curl -s "https://analytics.dugganusa.com/api/v1/stix-feed?minimal=true" | \
jq -r '.objects[] | select(.type=="indicator" and .confidence >= 80) | .pattern' | \
grep -oP "(?<=ipv4-addr:value = ')[^']+(?=')" | \
sort -u > /etc/firewall/dugganusa-blocklist.txtecho "Blocked $(wc -l < /etc/firewall/dugganusa-blocklist.txt) malicious IPs"
```
Why: Firewalls don't care about MITRE ATT&CK mapping. They just need the IP. Minimal mode + jq extraction gives you a clean blocklist.
Churn Prevention: What We Learned
We analyzed 30 days of consumer behavior and found:
| Consumer Status | Count | % of Total |
|-----------------|-------|------------|
| Loyal (5+ days) | ~20% | Core users |
| Evaluating (2-4 days) | ~25% | Trying it out |
| New (1 day) | ~15% | Just discovered |
| Churned (>2 days silent) | ~40% | Left |
Speculated churn reasons:
1. Storage/Ingestion Limits (35%) - 3.3MB payload overwhelmed their systems
2. Evaluation Complete (30%) - 1-3 day trial, decided not to continue
3. No Delta Feed (20%) - Wasteful downloads every poll
4. Holiday/Budget Cycles (10%) - Seasonal patterns
5. STIX Parser Incompatibility (5%) - Needed CSV instead
Our response: These 4 optimization parameters directly address the top 3 churn reasons.
Quick Reference
| Parameter | Purpose | Savings |
|-----------|---------|---------|
| `?since=<ISO8601>` | Delta feed (only new) | ~90% bandwidth |
| `?minimal=true` | Strip verbose context | ~85% size |
| `?limit=N&offset=N` | Paginated ingestion | Controlled |
| `?min_confidence=N` | High-confidence only | Variable |
Combined example:
```bash
curl -s "https://analytics.dugganusa.com/api/v1/stix-feed?since=2025-12-21T00:00:00Z&minimal=true&limit=100"
```
API Reference
Base URL: `https://analytics.dugganusa.com/api/v1/stix-feed`
• `since` (ISO8601) - Return only indicators modified after this timestamp
• `minimal` (boolean) - Strip verbose context, return core fields only
• `limit` (integer) - Maximum number of indicators to return
• `offset` (integer) - Skip first N indicators (for pagination)
• `min_confidence` (integer) - Minimum confidence threshold (0-100)
• `days` (integer) - Return only indicators from last N days
• `unique_only` (boolean) - Return only DugganUSA unique discoveries
Response: STIX 2.1 JSON Bundle with pagination metadata
Conclusion
3.3MB per poll is wasteful. 15KB with delta + minimal is efficient.
We built the optimizations. Now use them.
Questions? [email protected]
Powered by DugganUSA LLC
Feed: https://analytics.dugganusa.com/api/v1/stix-feed
Status: https://status.dugganusa.com
Dashboard: https://analytics.dugganusa.com/dashboard
*"Your security is our problem now."*
Get Free IOCs
Subscribe to our threat intelligence feeds for free, machine-readable IOCs:
AlienVault OTX: https://otx.alienvault.com/user/pduggusa
STIX 2.1 Feed: https://analytics.dugganusa.com/api/v1/stix-feed
Questions? [email protected]
The cheapest, fastest, most accurate threat feed on the internet.
275+ enterprises pulling daily. 1M+ IOCs. 17.4M indexed documents. We beat Zscaler by 43 days on NrodeCodeRAT. Starter tier $9/mo — less than any competitor’s sales demo.
Look up an IOC → · Audit your brand on AIPM → · See pricing →
Was this useful? Thirty seconds, no cookies, no tracking, no third parties, your address hashed and never stored. If the box below does not load, the same question lives at https://analytics.dugganusa.com/nps.html?post=the-problem-3-3mb-every-hour-is-wasteful