Why does NiFi 1.9 on Kubernetes encounter FileSystemRepository archive constraints errors and how can it be resolved?

Below is some guidance that might help you narrow down the root cause on your Kubernetes deployment. Note that while this error originates in Apache NiFi’s internal FileSystemRepository (its archive cleanup mechanism), when running in Kubernetes you also need to ensure that persistent storage is configured correctly.

What the Error Means

The error

  o.a.n.c.repository.FileSystemRepository Unable to write to container default due to archive file size constraints; waiting for archive cleanup

suggests that NiFi’s mechanism for archiving the flow configuration hit its set limits. Even after increasing the archive storage size or disabling it, the issue reoccurs after some time. That indicates that the archive or cleanup process is not keeping pace with new flow updates, which causes the repository to stall.

Areas to Investigate

1. Persistent Volume Provisioning

   On Kubernetes the NiFi container typically relies on a Persistent Volume (PV) for storing its flow configuration and archives. Verify that:

   • The PV is mounted correctly into the NiFi container.
   • There is sufficient disk space available on the PV.
   • The underlying storage class meets the performance and I/O requirements for frequent file updates and deletions.

2. Disk I/O and Filesystem Performance

   Issues related to file writes and deletions might also be influenced by I/O latency or unusual filesystem behavior. Since the problem appears after about 30 minutes, check whether:

   • The disk subsystem is experiencing delays.
   • There are any Kubernetes events or node-level alerts related to disk pressure or I/O throttling.

3. NiFi Internal Cleanup Process

   Although you’ve adjusted the archive settings, the cleanup process itself could be delayed or blocked. Monitor the NiFi logs for any additional errors or warnings related to cleanup tasks, thread contention, or related resource constraints.

4. Custom Processors

   While it appears less likely, custom processors might be contributing to an increased frequency of configuration changes or archiving activity. Double-check that none of these processors are overriding or interfering with standard repository behavior.

Next Steps

• Review your Persistent Volume Claim (PVC) and PV settings: Ensure they provide adequate capacity and performance.
• Look into NiFi logs: There may be additional clues if there are errors related to the cleanup threads or disk I/O.
• Monitor Node & Pod Metrics in Kubernetes: Check if there are any alerts on disk utilization or node-level resource constraints.
• Test with a Temporary “Simpler” Flow: This can help determine if the archive issue scales with your flow’s activity or configuration changes.

Conclusion

The message is typically a server (NiFi node)–level issue relating to how NiFi manages its flow configuration archive – not an indication of a misbehaving processor on its own. On Kubernetes, pay particular attention to how storage is provisioned and its performance characteristics. The issue isn’t directly a Kubernetes configuration error but rather an interplay between the NiFi archive settings and the characteristics of the persistent storage used by your Kubernetes deployment.

If after these checks the problem persists, consider reaching out to the Apache NiFi community for deeper insight into the archive cleanup mechanism.