AWS Kinesis Streaming

Real-time streaming — Data Streams (Shards), Firehose (S3 delivery), Analytics (SQL on stream).

AWSAdvanced

Why

Kinesis ingests millions of events/sec — logs, clicks, IoT. Firehose delivers to S3/Redshift with batching.

Analytics runs SQL on stream. Unlike SQS, data retained for replay.

When

Use for real-time analytics, log ingestion, IoT, and clickstreams. Avoid for simple async work (use SQS) or batch (use S3 + Lambda).

How

1. Stream — aws kinesis create-stream --shard-count 5 2. Put — aws kinesis put-record --stream-name my --data <base64> --partition-key user123 3. Consume — KCL or Lambda trigger, processes shard sequentially 4. Retention — 24h default, up to 365d, replay old data 5. Firehose — auto-deliver to S3/Redshift/OpenSearch, batching + transform Lambda 6. Analytics — SQL on stream, tumbling windows, aggregates 7. Sharding — partition key → hash → shard, scale by adding shards 8. Enhanced fan-out — 2MB/sec per consumer, no contention

Example

# Create stream + Firehose to S3 aws kinesis create-stream --stream-name events --shard-count 5 aws firehose create-delivery-stream \ --delivery-stream-name events-to-s3 \ --delivery-stream-type KinesisStreamAsSource \ --kinesis-stream-source-configuration KinesisStreamARN=arn:aws:kinesis:... \ --s3-destination-configuration RoleARN=...,BucketARN=...,PrefixingFormat=events/ // Lambda consumer export const handler = async (event) => { for (const record of event.Records) { const data = Buffer.from(record.kinesis.data, "base64").toString(); await processEvent(JSON.parse(data)); } };

Common Problems

⚠ProvisionedThroughputExceeded — add shards or use enhanced fan-out
⚠Replay needs retention — default 24h, increase if you need to reprocess
⚠Hot shard — one partition key dominates, use better partition key distribution

Best Practices

✓Use Firehose for S3 delivery — auto-batches, transforms, no consumer code
✓Enhanced fan-out for multiple consumers — 2MB/sec each, no shared capacity
✓Choose partition key wisely — distribute evenly, avoid hot shards

FAQ

What is AWS Kinesis Streaming and what problem does it solve?

Real-time streaming — Data Streams (Shards), Firehose (S3 delivery), Analytics (SQL on stream). It lets developers deploy, store, and scale without self-managing infrastructure — a core reference for cloud service selection.

How do I get started with AWS Kinesis Streaming in 5 minutes?

Two steps to run it: (1) Stream — aws kinesis create-stream --shard-count 5; (2) Put — aws kinesis put-record --stream-name my --data <base64> --partition-key user123. Validate on the free tier first, then wire into production.

What is the #1 pitfall with AWS Kinesis Streaming?

The most frequent issue is: ProvisionedThroughputExceeded — add shards or use enhanced fan-out. Run a pre-launch checklist item by item, and add policy/config scanning in CI.

What are the best practices for AWS Kinesis Streaming?

Best practices for AWS Kinesis Streaming include: Use Firehose for S3 delivery — auto-batches, transforms, no consumer code; Enhanced fan-out for multiple consumers — 2MB/sec each, no shared capacity; Choose partition key wisely — distribute evenly, avoid hot shards. The first one — "Use Firehose for S3 delivery — auto-batches, transforms, no consumer code" — should be a team default baked into your IaC templates.

How do I control cost and avoid bill surprises with AWS Kinesis Streaming?

Three ways to prevent overspend: (1) tag every resource with env/team and enable billing alarms; (2) set budget thresholds to trigger Slack/email alerts; (3) review idle resources in Cost Explorer regularly and clean them up. Most AWS Kinesis Streaming options offer a free tier — validate within free quota before scaling.