Annotation, Drift, and the Hidden Geometry of AV LiDAR

Everyone talks about the models. PointPillars, CenterPoint, TransFusion, pick your architecture, throw a leaderboard number on it, ship a blog post. But if you’ve actually operated AV LiDAR pipelines in production, you know the model is maybe 20% of the problem.

The other 80%? Annotation quality assurance, ground truth validation, orientation math that breaks at edge cases, model drift that nobody notices until a safety team flags it six weeks later, and the human-in-the-loop workflows that keep the whole thing from quietly degrading.

This is what we do at HuSig. Not the flashy part, the part that determines whether your perception stack actually works when it matters.

The Annotation Problem Nobody Wants to Talk About

LiDAR annotation looks simple on the surface: draw 3D bounding boxes around objects in a point cloud. In practice, it’s a nightmare of ambiguity.

Consider a parked SUV partially occluded by a delivery van. Your annotator sees maybe 40% of the point cloud surface. They need to estimate the full extent of the vehicle, length, width, height, from a sparse, one-sided slice of reflected laser returns. They need to nail the orientation (yaw) to within a few degrees, because even small angular errors cause 3D Intersection over Union (IoU) to collapse. A 5-degree yaw deviation on a 4.5-meter sedan can drop your 3D IoU from 0.85 to below 0.6, which most benchmarks would classify as a missed detection entirely.

Then multiply this by tens of thousands of frames, across dozens of annotators with varying experience levels, working across time zones.

The quality problem isn’t “are the boxes there.” It’s “are the boxes geometrically precise enough to train a model that won’t hallucinate phantom vehicles or miss real ones at highway speed.”

The Math That Keeps Us Honest

Let’s get into the geometry for a moment, because this is where most teams either get it right or silently accumulate error.

3D IoU is the standard metric for evaluating bounding box quality, but computing it for oriented (rotated) boxes in 3D is non-trivial. Unlike axis-aligned 2D IoU, which is just a min/max operation, 3D IoU for arbitrarily oriented boxes requires computing the intersection volume of two rotated cuboids. The typical approach decomposes this into a bird’s-eye-view (BEV) polygon intersection (using the Sutherland-Hodgman algorithm or convex hull clipping) multiplied by the overlap along the vertical axis.

Floor estimation is another area where small errors propagate. LiDAR returns from the ground plane need to be identified and removed before annotation and model inference. RANSAC-based ground plane fitting works in flat environments, but introduce a slope, a speed bump, or uneven terrain and your ground plane estimate shifts, which means every bounding box’s z-position and height are now biased. We’ve seen cases where a 10cm floor estimation error systematically shifted all pedestrian boxes, making them appear to float or sink into the road surface.

Orientation encoding is deceptively tricky. Yaw angles wrap around at ±π, creating discontinuities that confuse both annotators and regression losses. Some teams decompose heading into sine/cosine components; others use a discrete-continuous hybrid approach. The choice matters enormously for annotation consistency. If two annotators encode the same vehicle’s heading differently near the wraparound boundary, your ground truth contains contradictory labels that the model has to somehow reconcile during training.

Cuboid parameterization, the representation of a 3D box as (x, y, z, l, w, h, θ), hides another subtlety. The center of a LiDAR-observed object often falls in a sparse or empty region of the point cloud, because LiDAR only captures the front-facing surfaces. Annotators and models alike must extrapolate the center from partial observations, and systematic biases in this extrapolation directly impact downstream tracking and prediction modules.

MLOps for Perception: Drift Is the Silent Killer

Training a good perception model is hard. Keeping it good is harder.

Model drift in AV LiDAR manifests in ways that aren’t always obvious. A sensor gets recalibrated with slightly different extrinsics. The fleet expands to a new city with different road geometry and vegetation patterns. Seasonal changes shift the distribution of point cloud density, wet roads scatter returns differently, snow accumulates on vehicle roofs and changes their apparent dimensions.

The standard MLOps toolkit, monitoring loss curves, tracking aggregate mAP, catches catastrophic failures but misses the slow degradation that actually kills you. What you need is stratified performance monitoring: per-class, per-distance-band, per-occlusion-level metrics tracked over time, with automated alerting when any stratum drifts beyond a threshold.

We build these monitoring layers as first-class components, not afterthoughts. That means embedding evaluation pipelines directly into the data ingestion workflow, so that every new batch of annotated data is immediately scored against the current model’s predictions. If the model suddenly struggles with cyclists at 40+ meters in a specific geographic region, we know within hours, not weeks.

Retraining triggers should be data-driven, not calendar-driven. We track distribution shift across point cloud statistics (density histograms, range distributions, class frequencies) and trigger retraining when the incoming data diverges significantly from the training distribution, measured by metrics like Population Stability Index or KL divergence on binned feature distributions.

Human-in-the-Loop: Not a Fallback, a Feature

There’s a persistent myth in ML engineering that HITL is a temporary crutch. Something you tolerate until the model gets good enough to run autonomously. In safety-critical AV perception, this is exactly backwards.

HITL is a permanent, designed-in component of the pipeline. Here’s why.

Compliance and audit trails. Regulatory frameworks for autonomous vehicles increasingly require demonstrable human oversight of the perception training pipeline. This isn’t just “a human looked at it.” It’s traceable, timestamped review workflows where qualified reviewers validate annotation quality, flag edge cases, and sign off on dataset releases. Every annotation that feeds into a safety-critical model needs a chain of custody.

Edge case triage. The long tail of driving scenarios, construction zones, emergency vehicles with non-standard light patterns, pedestrians carrying large objects that distort their silhouette, cannot be reliably handled by automated QA alone. Human reviewers with domain expertise identify these cases, route them to specialized annotation workflows, and ensure they’re adequately represented in the training set.

Consensus-based quality scoring. For ambiguous annotations (and in LiDAR, ambiguity is the norm, not the exception), we implement multi-annotator consensus workflows. The same frame gets independently annotated by multiple reviewers, and disagreements are algorithmically detected and escalated. The consensus score becomes a confidence weight during training. Uncertain annotations contribute less to the loss function, preventing the model from overfitting to noisy labels.

Active learning loops. HITL isn’t just about validation. It’s about strategically selecting which data to annotate next. By routing model-uncertain predictions to human reviewers, you focus annotation effort where it has the highest marginal value. This cuts annotation costs significantly while improving model performance faster than random sampling ever could.

What This Looks Like in Practice

A typical engagement with us starts with an audit. We look at your existing annotation pipeline, your QA metrics (or lack thereof), your model evaluation framework, and your retraining cadence. More often than not, we find that teams have invested heavily in model architecture but underinvested in the data infrastructure that feeds it.

From there, we build what’s needed: automated ground truth validators that catch geometric inconsistencies before they reach the training set, monitoring dashboards that surface drift at the stratum level, HITL workflows that balance thoroughness with throughput, and the connective tissue between all of these, the MLOps pipelines that make the whole system self-correcting rather than reliant on periodic manual intervention.

The unsexy truth about AV perception is that the teams shipping the most reliable systems aren’t necessarily using the fanciest architectures. They’re the ones who’ve invested in annotation quality, geometric validation, continuous monitoring, and principled human oversight.

That’s the work. And it’s the work that actually matters.