How can I convert a YOLOv8-OBB model’s multiarray output to separate class labels and coordinates in Xcode?

Below is an overview of the situation and some suggested steps to take:

---

Why You’re Seeing a MultiArray

Core ML treats your YOLOv8n-OBB model as an object detector rather than a classifier. In typical image classifiers built with Core ML, outputs are automatically reinterpreted into a dictionary of class labels and confidences. In contrast, custom detectors (and even many YOLO-style models) export their raw output as a multiarray. This multiarray contains prediction data (bounding box coordinates, confidence scores, etc.) that require post‑processing (including non‑maximum suppression and coordinate conversion) to yield “human‑friendly” results.

---

Why Converting with a classifier_config Didn’t Work

Using the classifier conversion API (and providing a classifier_config with labels) works only when the model “naturally” produces a probabilities output tensor whose size matches the number of classes. With object detectors like YOLOv8n‑OBB, the number of output values (for bounding boxes and associated scores) does not match a simple list of class probabilities. That’s why you get errors like “number of classes must match the size of the tensor corresponding to ‘probabilities’.”

---

Recommended Approaches

1. Implement Custom Post‑Processing in Your App

   The most straightforward solution is to leave the Core ML model as it is (with its multiarray output) and perform post‑processing in Swift. For example, you can parse the MultiArray to:

   • Extract bounding box coordinates (which for OBBs might include rotation or additional parameters),
   • Pull out the corresponding confidence scores,
   • Apply non‑maximum suppression (if needed).

   This approach means writing a custom function that interprets the array according to the model’s output design. You can often base your logic on how YOLO decodes the multiarray into predictions.

2. Use Vision for Wrapping Your Model

   In some cases it may be beneficial to wrap your Core ML model within a Vision request. The Vision framework can help manage image preprocessing and, if needed, can also be used to post‑process outputs (though for custom detectors you will still need to implement the parsing logic).

3. Exploring a Custom MLModel with Pre/Post‑Processing Layers

   If you want the model’s preview in Xcode (which typically expects a classification model with separate confidence and label outputs), you must package your post‑processing logic inside the model. Using custom layers in your Core ML model (or creating a model “wrapper” with an additional post‑processing pipeline) may work—but it is more involved. Core ML Tools allows you to add custom layers, but that means replicating non‑maximum suppression and coordinate transformation steps inside your model graph.

---

Example: Post‑Processing in Swift

Below is a highly simplified code snippet illustrating how you might post‑process a multiarray output:

SWIFTCopied
import CoreML

func parseYOLOOutput(_ multiArray: MLMultiArray) -> [(boundingBox: CGRect, confidence: Float)] {
    // Assume the output multiarray is flattened to specific values per prediction.
    // You’ll likely need to rearrange the array according to your model.
    let predictionCount = multiArray.count / numberOfValuesPerPrediction // adjust accordingly
    var detections: [(CGRect, Float)] = []
    
    for i in 0..<predictionCount {
        let baseIndex = i * numberOfValuesPerPrediction
        // For example, assume the next 4 values represent x, y, width, height of the bounding box,
        // and the last is the confidence score.
        let x = Float(truncating: multiArray[baseIndex])
        let y = Float(truncating: multiArray[baseIndex + 1])
        let width = Float(truncating: multiArray[baseIndex + 2])
        let height = Float(truncating: multiArray[baseIndex + 3])
        let confidence = Float(truncating: multiArray[baseIndex + 4])
        if confidence > threshold { // define your threshold
            let bbox = CGRect(x: CGFloat(x), y: CGFloat(y), width: CGFloat(width), height: CGFloat(height))
            detections.append((bbox, confidence))
        }
    }
    return detections
}

Note: You’ll need to know exactly how your YOLO model orders its output values (especially for OBBs) to correctly parse them. The above example assumes a simple structure that may require adjustment based on your model’s design.

---

Summary

• A multiarray output is expected for custom object detectors and does not automatically separate class labels from coordinates.
• The classifier conversion API is meant for models whose outputs represent class probabilities—not for detectors.
• The two practical options are to perform post‑processing in your iOS app (using Swift, possibly wrapped in Vision) or to build a custom Core ML model that embeds the post‑processing logic.

By using one of these approaches, you’ll be able to extract the confidence values and coordinates (oriented bounding boxes) from the raw multiarray and then render or further use them in your iOS app.

If you need further guidance on integrating Core ML models into Xcode or writing custom post‑processing code, please refer to the “Integrating a Core ML Model into Your App” documentation.