Phase 1: The Naive HTTP Polling

Repository: Source Code on GitHub
Load Test Report: Download/View k6 HTML Report

The Baseline: Understanding the Cost of Overhead

Every evolution begins with a baseline. In Phase 1, we implement the most common architectural pattern for data ingestion: The RESTful POST Pipeline. While intuitive and easy to debug, this phase serves as a controlled environment to witness the inevitable collapse of traditional I/O under high-concurrency pressure.

The Architectural Approach

In this "Naive" stage, we treat each GPS coordinate as an independent HTTP transaction. The device acts as a client, initiating a new request for every movement.

• Protocol: HTTP/1.1
• Method: POST
• Payload: Standard JSON
• Server Stack: Node.js (Express) + Basic Validation Middleware

The "Naive" Implementation

The logic is straightforward: The server exposes an endpoint, validates the incoming JSON, and immediately attempts to persist the data.

"Simplicity is a virtue, but in high-performance systems, naive simplicity is a liability."

Data Ingestion Flow (Logic)

1. Receive POST request.
2. Authenticate and Validate Schema.
3. Synchronous Database Write (Blocking).
4. Return 201 Created.

import { Request, Response } from 'express';
import { Database } from '../../infrastructure/db';
import { validateGpsSchema } from '../../domain/validators';
 
export class NaiveIngestionController {
  
  /**
   * THE BOTTLENECK:
   * Handling individual GPS points via standard HTTP POST.
   * This forces a full TCP handshake and DB lock for every coordinate.
   */
  static async handleIncomingPoint(req: Request, res: Response) {
    try {
      const payload = req.body;
 
      // 1. Schema Validation Overhead (CPU Bound)
      const { isValid, errors } = validateGpsSchema(payload);
      if (!isValid) {
        return res.status(400).json({ status: 'rejected', errors });
      }
 
      // 2. Direct Database I/O (The Fatal Flaw)
      // Writing directly to disk for a single coordinate without buffering.
      // At 1M req/sec, this will instantly exhaust connection pools and disk IOPS.
      await Database.query(
        `INSERT INTO fleet_locations (device_id, lat, lng, speed, timestamp) 
         VALUES ($1, $2, $3, $4, $5)`,
        [payload.deviceId, payload.lat, payload.lng, payload.speed, payload.timestamp]
      );
 
      // 3. HTTP Response Overhead
      // Sending back headers and JSON for every single point wastes bandwidth.
      return res.status(201).json({ 
        status: 'acknowledged', 
        latency: 'simulated-high' 
      });
 
    } catch (error) {
      // Unhandled exceptions in high-concurrency loops lead to memory leaks.
      return res.status(500).json({ status: 'system_failure' });
    }
  }
}

import { z } from 'zod';
 
/**
 * Domain Schema: Absolute strictness at the edge.
 * We reject any anomaly before it touches the event loop core.
 */
const GpsPointSchema = z.object({
  deviceId: z.string().uuid({ message: "Invalid Device Identity" }),
  lat: z.number().min(-90).max(90),
  lng: z.number().min(-180).max(180),
  speed: z.number().nonnegative(),
  timestamp: z.number().int().positive() // Unix epoch for high-speed indexing
}).strict(); // Critical: Blocks unexpected fields to prevent memory bloat
 
export const validateGpsSchema = (payload: unknown) => {
  // safeParse prevents throwing exceptions which are costly in V8
  const result = GpsPointSchema.safeParse(payload);
  
  if (!result.success) {
    return { 
      isValid: false, 
      errors: result.error.flatten().fieldErrors 
    };
  }
  
  return { 
    isValid: true, 
    data: result.data // Strongly typed & sanitized payload
  };
};

"While tools like Zod provide robust type-safety at the edge, synchronous validation at 1,000,000 requests per second is a fatal CPU-bound operation that instantly locks the Node.js Event Loop. This is the exact architectural driver for our evolution in Phases 3 and 4: we offload all heavy validation to underlying C++ workers utilizing Protobuf's inherent type-safety. To achieve absolute control at scale, the Edge Ingestion layer must be reduced to a frictionless, zero-weight router." — BackendEvolution Architecture Principles

Identifying the Critical Bottlenecks

As we executed a precise Ramping Arrival Rate test using k6, pushing the simulated load to a spike of 10,000 Concurrent VUs, the architecture exhibited a catastrophic but expected collapse. The system processed 299,502 requests but dropped 226,997 iterations before they could even hit the network layer. We identified three primary I/O bottlenecks causing this failure:

1. The TCP Handshake Exhaustion

HTTP/1.1 overhead is massive for small payloads (GPS points). The constant cycle of SYN/ACK handshakes for every 100 bytes of data consumes more CPU cycles than the actual data processing.

2. Header Bloat & Context Switching

Standard HTTP headers (User-Agent, Content-Type, Auth-Tokens) often exceed the size of the GPS payload itself. Node.js spend significant resources parsing these headers, leading to high Context Switching in the Event Loop.

3. Database Locking & I/O Wait

Directly hitting the database for every request created a massive I/O Wait state. Without a buffering layer, the server's throughput is limited by the disk's IOPS, causing request queuing and eventual timeouts.

Performance Metrics (k6 Stress Test Results)

Phase 1 confirms that HTTP is an unacceptable transport layer for high-frequency telemetry at scale. The protocol overhead is non-negotiable. With a 15.11% failure rate and over 226k dropped iterations due to TCP exhaustion at just 10k connections, maintaining real-time absolute control is mathematically impossible here.

To reclaim stability and eliminate the handshake tax, we must establish persistent, stateful connections. We move to Phase 2: The Naive WebSocket.