Posted: Friday, 12 December 2025
ARIN

Executive Summary

On 2 December 2025, an IPv4 block 23.150.164.0/24, correctly allocated to the Original Customer, was inadvertently removed and reissued to the Requesting Customer during a 4.10 allocation process. This error stemmed from the current manual and partially offline 4.10 inventory process.

The incorrect state persisted until 9 December 2025, when the Original Customer reported the issue. ARIN restored the 23.150.164.0/24 to the Original Customer, issued a replacement /24 to the Requesting Customer, coordinated withdrawal of the incorrect route announcement, and notified the affected parties.

This incident highlights known weaknesses in ARIN’s current Internet Number Resources (INR) Inventory handling for 4.10 transition space and underscores the need to complete the transition to a fully automated, integrated online inventory architecture.

Incident Description

Following the current allocation process for 4.10 space, an RSD analyst:

• Relied on legacy/manual 4.10 inventory artifacts, including a flat file and sparse allocation spreadsheet
• Did not recognize indicators in ARIN Online showing that 23.150.164.0/24 was already allocated to the Original Customer
• Removed 23.150.164.0/24 from the Original Customer
• Reissued that same /24 to the Requesting Customer

As a result, the registration record and associated ROAs for the Original Customer were deleted in error, and the /24 appeared as allocated to the Requesting Customer in ARIN’s systems.

Customer Impact and Risk

• The /24 was removed from the Original Customer account and assigned to another organization.
• The ROA associated with the block was removed and had to be recreated after restoration.
• The block was announced by a third-party provider under the incorrect registration, introducing risk of routing conflict and confusion.
• The incorrect state persisted for approximately seven days before detection.
• The Original Customer reported the issue via Ask ARIN and a Help Desk call on 9 December 2025.

The customer has not provided a technical impact statement.

Timeline of Events

(All event times are ET – Eastern Time)

25–26 November 2025

• 25 November, 11:59 AM – 4.10 space request received from the Requesting Customer.
• 26 November, 6:25 AM – Ticket assigned to an RSD Analyst for processing.

2 December 2025 – Incident Occurs

• 12:10 PM – Ticket approved for issuance of 4.10 space by designated RSD Analyst.
• ~12:10–12:30 PM – In the process of fulfilling the 4.10 request, the designated Analyst:
  • Opened the e-black-book (an offline Excel-based inventory file, separate from the primary online inventory system), reviewed the existing 4.10 allocations, and selected 23.150.164.0 as the next available sparse entry.
  • Returned to the ARIN Online management application and queried for 23.150.164.0 based on the entry identified in the e-black-book. At this time the analyst did not recognize that the /24 was already allocated to the Original Customer.
  • Performed a block split and deleted 23.150.164.0/24 – not recognizing that it was allocated to the Original Customer – which removed associated registry services (ROAs, reverse DNS, etc.).
  • Issued 23.150.164.0/24 to the Requesting Customer.

2–9 December 2025 – Incorrect State Persists

• The /24 remained misregistered.
• The Requesting Customer upstream provider announced the block.
• No automated detection of the error occurred.

9 December 2025 – Detection and Resolution

• 10:12 AM – The Original Customer submitted an Ask ARIN ticket regarding the problem.
• 10:14 AM – The Original Customer contacted the Help Desk; escalation to Director at 10:20 AM.
• 10:20–10:30 AM – Director reviewed block history and directed corrective actions.
• 10:30 AM – Director and CXO approved:
  • Removal of the /24 from the Requesting Customer
  • Issuance of a replacement /24 to the Requesting Customer
  • Restoration of 23.150.164.0/24 to the Original Customer
  • Coordination of route withdrawal
  • Update of inaccurate POC information
• 10:44 AM – First notification email sent to the Requesting Customer.
• 10:54 AM – Second email sent noting invalid phone contact.
• 12:01 PM – Corrective actions completed.

Root Cause

A manual 4.10 workflow that relies on a combination of online systems and offline flat files/spreadsheets for inventory management allowed a current customer allocation to be mistakenly identified as available for issuance. This reliance on offline spreadsheets is a legacy constraint where post-runout 4.10 inventory is maintained outside the primary online system to keep it reserved. The lack of a unified view of inventory and related business-rule-driven system controls enabled the error to proceed without detection.

Contributing Factors

• Hybrid inventory architecture (online + offline) for 4.10 space.
• Sparse allocation methods implemented through manual tools rather than integrated system logic.
• Generic warning messages that are not routing aware or business-rule driven.
• High demand on analysts to catch procedural errors in a manual “swivel chair” workflow.

Mitigation Plan and Next Steps

Immediate / Near-Term Controls (Completed)

• Updated Process Controls (completed)
  • RSD has implemented additional process controls that require a dual review for all ticketing type workflows that include a network delete.
  • Only a limited set of experienced analysts are permitted to perform this function.
  • Reviews and approvals are performed at set times each day with a second reviewer involved for any ticket that includes a delete step.
• Updated 4.10 Issuing Playbook
  • Document and enforce a revised playbook for issuing 4.10 (with checklists) that includes:
    • Required checks for existing allocations and ROAs
    • Explicit verification steps prior to any delete/reissue action

System and Architecture Improvements (Medium-Term)

Accelerate the ongoing INR Inventory Management Roadmap item: This incident reinforces the urgency of the architecture work already underway (reviewed Oct 2025) to move legacy offline inventories into a modern, online architecture. Specific alignment actions include:

• Stronger business rule–based warnings and controls
  • Enhance warning logic when issuing or modifying 4.10 space to include:
    • Clear alerts if the /24 is already allocated to an Org
    • Clear alerts if active ROAs exist for the exact block or covering prefix
  • Implement system controls for resource types and staff roles, with flags and audit trails for review and auditing.
  • Replace generic, non–ROA-aware warnings that are easily treated as noise.
• Continue Engineering solution for offline inventory
  • Move offline 4.10 and microallocation inventories, and the viip file for IPv6, into the integrated online inventory architecture.
  • Eliminate reliance on separate spreadsheets and flat files for production issuing.
  • Implement business-rule-driven warnings for existing allocations and ROAs
  • Introduce role-based controls, flags, and audit trails
• Advance the “Updated Resource Status Taxonomy” work
  • Ensure 4.10 status and history are fully visible and consistent inside the primary system.
  • Provide analysts with a clear, unified view of current holder, status, and ROA/IRR context.
• Fast-track automation of all inventory issuing
  • Reduce or eliminate manual issuing where possible, with priority for higher-risk categories such as 4.10 and RPKI-covered space.

Regards,

American Registry for Internet Numbers (ARIN)