Streaming Problems and How They Are Fixed

A streaming platform like Apache Kafka stores data as a commit log — an append-only, numbered file of records that readers walk forward through, keeping their own position rather than deleting what they read. That design is what makes it powerful, but it is also a distributed, stateful, disk-backed system, and most of the pain of running one comes from a small, predictable set of failures. This day walks through each of them with a concrete worked example, why it happens, and the fix.

A few terms recur, so pin them down once here and each section will re-gloss the ones it leans on. A partition is one independent append-only sub-log; a topic is split into several so many machines can serve it in parallel, and records are strictly ordered within a partition but not across partitions. An offset is a record's position in a partition — just a counter, 0, 1, 2, and up. A consumer reads records; a consumer group is a set of consumers that split a topic's partitions between them so each partition is read by exactly one member. A producer writes records. Almost every failure below is a variation on one theme: readers cannot keep pace with writers, or the group cannot agree on who reads what.

The sections are ordered roughly by how often each one bites in practice. The single most common is the first: consumers falling behind.

Rule of thumb: nearly every streaming incident is a pacing or a coordination problem — either the readers drifted behind the writers, or the group spent its time reshuffling instead of reading.

Consumer lag is the gap between the newest record written to a partition and the record a consumer group has finished processing — in plain terms, how far behind real time your readers are. Small, steady lag is normal; lag that grows means writers are outpacing readers and the system is silently drifting further behind every second.

Worked example: a topic takes writes at 50,000 messages per second at peak. Your consumer group, as currently sized, drains 30,000 per second. Every second the backlog grows by 20,000 records. After ten minutes of peak traffic the group is 12 million records behind — and because the reader is processing minutes-old data, every downstream effect (an email, a dashboard, a fraud check) is that stale too.

Lag is a simple subtraction, arrows aside:

lag = newest_offset - committed_offset
growth_rate = produce_rate - consume_rate   (here 50k - 30k = +20k/s)

Why it happens: consumer throughput has a ceiling set by three things — how many consumers can work in parallel (capped by the partition count, since one partition is read by one group member), how much work each record costs (a slow database write per record kills throughput), and how efficiently each poll fetches data. When produce rate crosses that ceiling, lag grows without bound.

How it is fixed:

• Add parallelism. Raise the partition count and add consumers up to that count so more records are processed at once — the group's throughput ceiling is the partition count, so eight partitions allow at most eight busy consumers.
• Do less per record. Batch external writes (one bulk insert per poll instead of one insert per record), cache lookups, and move slow side-effects off the hot path.
• Tune fetching. Let each poll pull more data at once so the consumer spends time processing, not waiting on the network.

max.poll.records=1000
fetch.min.bytes=1048576
fetch.max.wait.ms=100

• Autoscale on lag, not CPU. Scale the consumer count up when lag rises and down when it drains — lag is the true demand signal, CPU is a proxy.
• Monitor lag per group per partition. This is the top-line health metric; alert on growing lag before the backlog becomes hours deep.

kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group orders-consumer

Rule of thumb: watch lag as your primary health signal, size partitions and consumers for peak produce rate with headroom, and cut the per-record work before you add machines.

A rebalance is the process where a consumer group reassigns its partitions among members — it runs whenever a consumer joins, leaves, or is presumed dead. The classic (eager) rebalance is stop-the-world: every consumer stops reading, the group agrees on a new assignment, then all resume. A rebalancing storm is when this fires over and over, so the group spends its time reshuffling instead of processing.

Worked example: a group of ten consumers processes an image-heavy stream. One consumer hits a slow batch and does not send its periodic heartbeat (the "I'm alive" ping) within the timeout. The group coordinator (the broker tracking membership) declares it dead and triggers a stop-the-world rebalance — all ten stop. During the pause, lag spikes; when reading resumes, a second consumer is now behind and slow, misses its heartbeat, and the group rebalances again. Each rebalance costs a few seconds of zero progress; ten in a row and the group makes almost no forward progress for minutes while lag climbs.

Why it happens: two different timeouts govern liveness, and confusing them causes storms. session.timeout.ms is how long the coordinator waits for a heartbeat before declaring a consumer dead. max.poll.interval.ms is how long a consumer may spend processing between polls before the group assumes it is stuck. A consumer that processes a big batch slowly can blow past either one, get evicted, rejoin, and trigger a fresh rebalance — and with eager rebalancing, one flapping member stalls everyone.

How it is fixed:

• Cooperative (incremental) rebalancing. Instead of stop-the-world, only the partitions that actually need to move are revoked; everyone else keeps reading. One member joining or leaving no longer pauses the whole group.

partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

• Static group membership. Give each consumer a stable identity (group.instance.id) so a quick restart or deploy does not look like a departure-and-arrival that forces reassignment.
• Tune the timeouts to reality. Set max.poll.interval.ms above your worst-case batch processing time, and keep session.timeout.ms comfortably above the heartbeat interval so a brief GC pause is not read as death.
• Poll smaller batches. Fewer records per poll means each processing cycle finishes well inside the interval, so the consumer keeps checking in.

max.poll.records=200
max.poll.interval.ms=300000
session.timeout.ms=45000
heartbeat.interval.ms=3000

Rule of thumb: use cooperative rebalancing plus static membership so routine restarts do not pause the group, and size the poll batch so processing always finishes before the interval expires.

A record's partition is chosen by hashing its key (a field like an account id or a country id) modulo the partition count, so all records with the same key land on the same partition and stay ordered. A hot or skewed partition is one that receives a wildly disproportionate share of the traffic because the key is unevenly distributed — one consumer drowns while its peers idle, and adding consumers does not help because a single partition is still read by a single group member.

Worked example: an events topic is keyed by country_id across ten partitions. Roughly 70% of traffic is from one country, so 70% of every record hashes to one partition. That partition's consumer must handle 70% of the load alone; the other nine consumers split the remaining 30% and sit nearly idle. Total throughput is capped by what that one busy consumer can do — the topic behaves as if it barely has parallelism at all, even though nine machines are provisioned.

Why it happens: the partitioner distributes keys evenly, not traffic. If a handful of key values carry most of the volume, the hash spreads those few keys across few partitions and the load follows. More partitions or more consumers cannot rescue a key space where one value dominates.

How it is fixed:

• Pick a higher-cardinality key. Key by something with many roughly-equal values (a user_id, a session_id) instead of a coarse bucket like country, so load spreads across all partitions.
• Salt the key (compound key). When you must key by the skewed field, append a small random or round-robin suffix — country_id:0 through country_id:7 — turning one hot key into several. The catch: this breaks strict per-key ordering, so only salt when you do not need all of one country's events strictly ordered.

hot key:   "US"            lands on one partition
salted:    "US:0".."US:7"  spreads over up to 8 partitions

• Redesign what "in order" means. Often you only need per-user order, not per-country order; keying by the finer entity both fixes the skew and keeps the ordering that actually matters.

Rule of thumb: check that no single key value carries most of the traffic; if one does, key by a finer-grained field or salt the hot key, and only preserve the ordering granularity you truly need.

The partition count is chosen when a topic is created and is awkward to change: raising it later reshuffles which keys land where (breaking per-key ordering for existing keys), and Kafka cannot shrink a topic's partition count at all. So the number you pick tends to be the number you live with, and both extremes hurt.

Worked example: over-provisioning — a team creates a topic with 1,000 partitions "to be safe" for a stream that needs maybe 20. Every partition costs open file handles, memory, and replication traffic on the brokers, and the controller must track each one in cluster metadata. With replication factor 3 that is 3,000 partition replicas the controller manages; a broker restart or leader election now churns through thousands of assignments, lengthening recovery and slowing metadata operations across the whole cluster. Under-provisioning is the mirror image: a topic created with 4 partitions can never run more than 4 consumers in a group, so when traffic triples there is no way to add parallelism without a disruptive repartition.

Why it happens: partitions are both the unit of parallelism (more partitions allow more consumers) and a per-unit cost on the brokers and controller. Optimizing only for future parallelism inflates the count; optimizing only for today's load starves it later. Neither the metadata cost nor the ordering-reshuffle of a later change is visible at creation time.

How it is fixed:

• Capacity plan from throughput. Estimate peak produce rate and the per-partition throughput a single consumer sustains; partition count is roughly peak demand divided by per-consumer capacity, rounded up.
• Over-partition modestly, not wildly. Leave headroom for a few years of growth (a small multiple), not an imagined 50x — enough that you rarely need to change it, cheap enough that the controller is not burdened.
• Be key-space aware. More partitions only help if the key has enough distinct values to fill them; 1,000 partitions with 10 hot keys just gives you 990 empty partitions and 10 hot ones.

Rule of thumb: size partitions to peak throughput with modest headroom, remember you can grow but never shrink the count, and never add partitions the key space cannot actually spread traffic across.

Kafka's default delivery guarantee is at-least-once: a record is never lost, but it may be processed more than once. The reason is offset commits — a consumer records progress by committing "I have processed up to offset N," and if it crashes after doing the work but before committing, the work is redone on restart. Out-of-order effects come from the flip side: records for the same entity landing on different partitions, which have no ordering relative to each other.

Worked example: a consumer reads order #900 at offset N, charges the customer's card, then — before it commits offset N — the process is killed (a deploy, an out-of-memory kill). On restart the group re-reads from the last committed offset, which is still N, so order #900 is processed again and the card is charged twice. Nothing was lost; the record was simply delivered a second time because the commit never landed.

Why it happens: processing and committing are two separate steps, and a crash can fall between them. Commit-after-processing (the safe default) means a gap there causes a duplicate; commit-before-processing means a gap causes a skip (data loss). You cannot make the two steps atomic against an outside system, so some redelivery is inherent.

How it is fixed:

• Idempotent consumers. Make processing safe to run twice for the same record. Track a stable business id (the order id, an event id) in the destination and ignore a record you have already applied — a dedupe key. Charging by INSERT ... ON CONFLICT DO NOTHING on the order id makes a second delivery a no-op.

dedupe: if seen(order_id) then skip, else apply and record order_id

• Per-key ordering via partitioning. To keep one entity's events in order, key by that entity so they all share a partition (which is strictly ordered). Out-of-order processing is usually a symptom of the wrong key spreading one entity across partitions.
• Offset-commit discipline. Always commit after the work is durably done, commit in the right place (not on an auto-timer that can fire mid-work), and prefer manual commits for anything with side effects.

Rule of thumb: assume at-least-once, make every consumer idempotent on a stable id, key by the entity that must stay ordered, and commit offsets only after the work is safely done.

Kafka can offer exactly-once within Kafka through the idempotent producer (which discards duplicate appends) and transactions (which write to several partitions and commit consumer offsets atomically). The trap is assuming this extends to systems outside Kafka. A Kafka transaction can only cover Kafka's own reads and writes; the moment your consumer writes to an external database, a payment API, or an email service, that effect is outside the transaction's reach.

Worked example: a pipeline reads from an input topic, transforms each record, and writes the result to an external Postgres table — a read-process-write flow. You enable Kafka transactions expecting exactly-once. But the transaction only atomically ties the Kafka read offset to any Kafka writes; the Postgres insert is a separate action. If the consumer inserts into Postgres, then crashes before the Kafka transaction commits the offset, on restart it reprocesses the record and inserts into Postgres again — a duplicate row — because Kafka's transaction never governed Postgres in the first place.

Why it happens: a transaction is atomic only over the resources one transaction manager controls. Kafka's manager controls Kafka partitions and Kafka offsets — not a foreign database's commit. Two independent systems cannot be committed as one atomic step without a shared protocol they do not have.

How it is fixed:

• Idempotent writes / upserts on the sink. Give each record a stable id and write with an upsert (INSERT ... ON CONFLICT DO UPDATE) keyed on it, so reprocessing overwrites rather than duplicates. This is the workhorse fix.
• Transactional outbox on the sink side. When the sink is a database you own, write the business change and an "event" row in the same local database transaction, then relay the outbox to Kafka separately — the database's own atomicity replaces the cross-system guarantee you cannot get.
• Aim for effectively-once end to end. Combine Kafka's within-Kafka exactly-once with idempotent external writes; the effect is exactly-once even though delivery is at-least-once. There is no broker switch that makes an external side effect exactly-once — you engineer it at the sink.

Rule of thumb: treat Kafka's exactly-once as Kafka-only, and make every external write idempotent (upsert on a stable id) so redelivery cannot double-apply — that is how you get exactly-once effect across the whole pipeline.

A poison message is a single record a consumer cannot process — an un-deserializable payload, a corrupt field, a schema the consumer does not understand. Because a partition is strictly ordered and read position advances one record at a time, a poison message that makes the consumer throw and retry forever blocks the whole partition behind it: every good record after it waits indefinitely.

Worked example: one producer bug emits a record whose body is not valid JSON (or not valid Avro — a compact binary format). The consumer tries to deserialize it, throws, and — with naive error handling — retries the same offset in a tight loop. It never advances past that offset, so the thousands of perfectly good records behind it on that partition are never read. Lag on that one partition climbs steadily while the other partitions look fine, which makes the incident easy to misread.

Why it happens: a consumer's default reflex is to retry on failure, which is correct for transient errors (a momentary database blip) but catastrophic for a permanent one — the bad record will fail identically forever, so retrying it forever starves the partition. The consumer cannot tell "try again later" from "this will never work" without a policy.

How it is fixed:

• Dead-letter topic (DLQ). A DLQ is a separate topic where records that cannot be processed are parked. After a bounded number of retries, the consumer publishes the failing record (plus the error and its original offset) to the DLQ, commits past it, and keeps processing the good records. The bad ones are inspected and reprocessed out of band.
• Skip-and-log with an alert. For records safe to drop, log the payload and offset, emit a metric, and move on — but always alert, so a rising DLQ or skip rate surfaces the producer bug rather than hiding it.
• Validate at the edge. Reject malformed records where they enter the system (schema validation at the producer or an ingest gateway) so a poison message never reaches the ordered log in the first place.

Rule of thumb: never let a consumer retry a permanently-bad record forever — bound retries, route failures to a dead-letter topic and alert, and validate payloads at the edge so poison never enters the partition.

Records have a schema — a formal description of their fields and types, commonly written in Avro or Protobuf. Over time the schema must change: new fields, renamed fields, dropped fields. A compatibility break is a schema change that makes existing consumers unable to deserialize new records (or new consumers unable to read old records), and because producers and consumers deploy independently, one careless change can break every reader of a topic at once.

Worked example: an orders record has a required customer_email field that consumers read directly. A producer team, cleaning up, ships a new version that drops customer_email. Their producers deploy first and start emitting records without it. Every existing consumer still expects the field, fails to deserialize the new records, and — depending on error handling — either crashes or poison-blocks the partition. A single producer deploy has taken down every consumer of the topic.

Why it happens: a topic is a long-lived contract read by many independently-deployed services, but a schema change is made by one team in isolation. Dropping or renaming a field, or changing a type, silently violates what every reader assumed — and nothing stops the deploy unless something is enforcing the contract.

How it is fixed:

• A Schema Registry with compatibility rules. A Schema Registry stores every version of a topic's schema centrally and rejects a new version that would break readers. Backward compatibility means new consumers can read old data; forward compatibility means old consumers can read new data; full requires both. The registry refuses to register the incompatible change, so the producer deploy fails safely instead of the consumers failing in production.
• Additive-only changes. Add new optional fields; never remove or rename a field readers depend on, and never repurpose a field's meaning. Additive changes are inherently compatible.
• Defaults on every field. In Avro/Protobuf, give new fields a default value so a reader that does not know the field still deserializes the record cleanly — the mechanism that makes additive evolution safe in both directions.

Rule of thumb: put a Schema Registry in front of any topic more than one team reads, evolve schemas additively with defaults, and let the registry's compatibility check block a breaking change before it deploys.

A log keeps records after they are read, but not forever — retention deletes the oldest records once they age past a configured time or the partition exceeds a size limit. Retention expiry as data loss happens when a consumer stays down (or lags) longer than the retention window: the unread records at the tail of the log are deleted before the consumer ever reads them, and they are simply gone. Backpressure is the general act of slowing or pausing producers when consumers cannot keep up, so the backlog never outgrows what the system can hold.

Worked example: a topic is configured to retain records for 6 hours. A consumer group goes down for a deploy that goes wrong and stays offline for 8 hours. When it comes back, it tries to resume from its last committed offset — but the oldest 2 hours of records it had not yet read have already aged out and been deleted. Those records are unrecoverable; the group skips ahead to the earliest surviving offset, having permanently lost 2 hours of events. Nothing errored loudly; the data just quietly fell off the end of the log.

Why it happens: retention deletes by age or size regardless of whether anyone has read the records — it protects the disk, not the consumer. If the time a consumer can be behind exceeds the retention window, the log garbage-collects data out from under it. The failure is silent because deletion is normal, expected behavior.

How it is fixed:

• Size retention with headroom. Set the window comfortably longer than your worst realistic consumer outage plus recovery — if a bad deploy can keep a consumer down 8 hours, 6-hour retention is a data-loss bug waiting to happen; make it days.
• Tiered storage. Tiered storage offloads older log segments to cheap object storage (like S3) while keeping recent ones on the broker, so you can retain days or weeks without paying for broker disk — extending the safety window dramatically.
• Alert on lag against the window. Alert when a group's lag approaches the retention window (for example, lag older than half the retention), so you act before records are about to expire, not after.
• Backpressure at the source. When a backlog is genuinely unrecoverable in time, slow or pause producers (or shed load upstream) so incoming data does not push older unread data off the log — trading fresh throughput to avoid losing history.

Rule of thumb: retention must exceed your worst-case consumer downtime with margin; use tiered storage to make a long window cheap, alert on lag approaching the window, and apply backpressure before unread data ages out.

Separate from a rebalancing storm is a subtler trap: a consumer whose processing legitimately takes longer than max.poll.interval.ms — the maximum time a consumer may spend between polls before the group assumes it is stuck and evicts it. When honest work exceeds that limit, the consumer is kicked mid-batch, rejoins, gets the same records, and does it all again — a livelock where the system is busy but makes no forward progress.

Worked example: a consumer enriches each batch by calling a slow external service, and a batch of records takes 6 minutes to finish. But max.poll.interval.ms is the default 5 minutes. At minute 5, the coordinator sees no new poll, declares the consumer stuck, and evicts it — triggering a rebalance and reassigning its partitions. The consumer finishes its 6-minute batch and tries to commit, but it no longer owns those partitions, so the commit is rejected and the offsets never advance. It rejoins, is handed the same records, and starts the same 6-minute batch again — forever. The consumer is at 100% CPU and processing zero net records.

Why it happens: max.poll.interval.ms exists to detect a genuinely hung consumer, but it cannot distinguish "hung" from "doing slow but real work." If a single poll's worth of records takes longer to process than the interval allows, the safety mechanism misfires and evicts a healthy consumer, and the reprocessing loop never terminates.

How it is fixed:

• Raise max.poll.interval.ms above worst-case batch time. Give processing enough headroom that even the slowest realistic batch finishes before eviction.

max.poll.interval.ms=600000
max.poll.records=50

• Poll smaller batches. Fewer records per poll means each processing cycle is short and the consumer polls again well within the interval — often the cleanest fix.
• Offload slow work. Hand the slow per-record call to a worker pool or an async stage so the polling loop stays fast and only coordinates, rather than blocking on a 6-minute call.
• Pause and resume. For a deliberately long unit of work, use the consumer's pause/resume controls to keep the poll loop alive (heartbeating) while the heavy work proceeds, so the group never thinks the member died.

Rule of thumb: make sure the slowest batch you can produce finishes inside max.poll.interval.ms — shrink the batch, raise the interval, or offload the slow work — so honest slow processing is never mistaken for a dead consumer.

A single Kafka cluster lives in one region (one data-center footprint). If that entire region has an outage, the cluster — and every pipeline depending on it — goes down with it, and any data not replicated elsewhere is unreachable until the region recovers. Disaster recovery for streaming is the practice of keeping a usable copy of your topics in a second region so a regional failure is survivable.

Worked example: a company runs one Kafka cluster in a single cloud region. That region has a multi-hour outage. Producers cannot write, consumers cannot read, and the streaming backbone that feeds ordering, analytics, and fraud checks is simply offline for the duration — there is no second copy to fail over to, so recovery time is entirely at the mercy of the region coming back. Every downstream system that assumed the stream was always available stalls with it.

Why it happens: replication within a cluster (the in-sync replicas from the durability model) protects against losing a broker or a disk — it does not protect against losing the whole region those brokers share. Regional resilience is a different layer that a single cluster does not provide by default.

How it is fixed:

• Cross-cluster replication (MirrorMaker). MirrorMaker is Kafka's tool for copying topics from one cluster to another, typically in a second region. It continuously mirrors records so the standby region holds a current copy of the data.
• Offset translation. Because offsets differ between the two clusters, MirrorMaker also translates a consumer group's committed offsets from the primary to the replica, so consumers can resume near where they left off after a failover instead of restarting from zero or skipping data.
• Multi-region topics / active-passive. Run the second region as a warm standby (active-passive): normal traffic uses the primary, and on a regional outage you fail consumers and producers over to the replica. (Active-active — both regions taking writes — is possible but adds conflict and ordering complexity, so active-passive is the common starting point.)

Rule of thumb: in-cluster replication survives a broker; surviving a region needs cross-cluster replication with offset translation to a standby cluster, run active-passive so a regional outage is a failover, not an outage.

Everything above shares a root cause: a log platform is a real distributed system, and running one is not free. It is worth stepping back from specific failures to name the standing cost, because the sharpest fix for many "streaming problems" is realizing you did not need a streaming platform for that workload.

Running Kafka (or any Kafka-style log) means operating a fleet of brokers (the servers that store partitions), provisioning and monitoring their storage (partitions consume disk, file handles, and replication bandwidth), standing up monitoring (lag, rebalances, ISR health, disk, throughput — none of it optional), and carrying an on-call burden for a stateful system with its own tuning, capacity planning, and version-upgrade dance. That is a permanent team cost, paid whether or not the platform is heavily used.

That cost is justified when you genuinely need what a log uniquely provides — replay of history, fan-out of the same stream to many independent consumers, or per-key ordering at high throughput. It is not justified when a simpler tool fits:

• A plain queue (SQS, RabbitMQ classic) is simpler and cheaper when each message is a one-off job done once and never replayed — no partitions, no offsets, no consumer-group choreography.
• A database is the right tool when you need to look up and update current state by key, not append and replay a history — a log is an append-only history, not a place to mutate a row.
• A managed streaming service (Kinesis, Google Pub/Sub, or a hosted Kafka) keeps the log's capabilities while handing the broker operations, storage, and on-call to a provider — the fix when you need streaming but not the burden of running it.

Rule of thumb: a log earns its operational cost only when you truly need replay, fan-out, or high-throughput ordering; if each item is a do-once job use a queue, if you need current state use a database, and if you need the log but not the ops use a managed service.

This table collapses the day into problem-to-fix pairs — the fast reference for what breaks and what to reach for.

• Nearly every streaming failure is a pacing or coordination problem: readers drifted behind writers, or the group spent its time reshuffling instead of reading.
• Consumer lag is the top-line health signal — watch it per group per partition, alert on growing lag, and fix it with parallelism, per-record batching, fetch tuning, and lag-based autoscaling.
• Rebalancing storms come from members flapping and pausing the whole group; cooperative rebalancing plus static membership and sane timeouts keep routine restarts from stalling everyone.
• Hot partitions come from a skewed key, not too few consumers — spread load with a higher-cardinality key or a salted key, and only preserve the ordering granularity you actually need.
• Partition count is set once, can grow but never shrink, and costs the controller at the high end while capping parallelism at the low end — plan to peak with modest headroom.
• Delivery is at-least-once, so make consumers idempotent on a stable id, key by the entity that must stay ordered, and commit offsets only after the work is durably done.
• Kafka's exactly-once is Kafka-only; make external writes idempotent upserts (or use a transactional outbox) to get exactly-once effect across the whole pipeline.
• A single poison message retried forever blocks its partition — bound retries, route failures to a dead-letter topic and alert, and validate payloads at the edge.
• Schema changes break independently-deployed consumers; a Schema Registry with compatibility rules plus additive-only changes with defaults keeps a topic readable as it evolves.
• Retention deletes unread data once it ages out, so retention must exceed worst-case consumer downtime — extend it cheaply with tiered storage and alert on lag approaching the window.
• Honest slow processing that exceeds max.poll.interval.ms causes a reprocessing livelock — shrink the batch, raise the interval, or offload the slow work so a healthy consumer is never mistaken for a dead one.
• In-cluster replication survives a broker but not a region; cross-cluster replication with offset translation to an active-passive standby is what makes a regional outage a failover.
• A log platform is expensive to run — reach for it only when you truly need replay, fan-out, or high-throughput ordering, and use a queue, a database, or a managed service otherwise.