Determining SpriteKit Interaction using SKRenderer

Hey,

Wondering how other developers have been able to determine the location of a mouse event or tap (ie NSEvent) when using MetalView (MKTView) with SKRenderer with a SpriteKit scene (.sks scene) for a 2D game.

In the original scenario with SpriteKit, we could use SKViews convertPoint(fromView:) to determine where in the scene the user tapped. But with the SKRenderer we can no longer use convertPoint(fromView:) as its reliant on SKView being used and thus its making it difficult to determine.

What I do have is:

locationInWindow: NSPoint for showing me where in the MKTView which was touched

Any ideas, would be great

Many thanks

Answered by DTS Engineer in 917241022

The locationInWindow you already have is the right starting point. SKView can convert points for a scene it isn't presenting. The documentation for convert(_:to:) says it "performs the coordinate conversion as if the scene is presented inside the view." Keep an SKView that you never add to a window, give it the bounds of your MTKView, and convert the event location through it:

final class GameView: MTKView {
    var spriteRenderer: SKRenderer?

    // Never added to a window. It only converts points.
    private let converter = SKView(frame: .zero)

    override func mouseDown(with event: NSEvent) {
        guard let scene = spriteRenderer?.scene else { return }
        converter.frame = bounds
        let viewPoint = convert(event.locationInWindow, from: nil)
        let scenePoint = converter.convert(viewPoint, to: scene)
        for node in scene.nodes(at: scenePoint) {
            print(node.name ?? "unnamed", scenePoint)
        }
    }
}

Use bounds for the converter's frame rather than drawableSize. The view point from the event and bounds are both in points. The drawableSize documentation says its default value "is derived from the current view's size, in native pixels."

I tested this on macOS 27.0.1 by rendering marker sprites with SKRenderer at a backing scale factor of 2. Then I converted each sprite's rendered location with convert(_:to:) and compared the result with the sprite's position in the scene. The results matched exactly for the fill, aspectFill, and aspectFit scale modes, with two anchor points, three view aspect ratios, and an SKCameraNode. The nodes(at:) method found the expected sprite at every converted point. You can run the same check in your scene with the known-points technique from TN3124: Debugging coordinate space issues. Put a sprite at a known scene position, click it, and confirm that the converted point matches.

Converting Coordinate Spaces suggests location(in:) on NSEvent for mouse events in macOS, but it doesn't do this conversion for a scene that SKRenderer draws. The location(in:) documentation describes the node parameter as "A node that is a descendant of a scene presented in the window that received the mouse event." In my tests, it returned the point in view coordinates unchanged.

The locationInWindow you already have is the right starting point. SKView can convert points for a scene it isn't presenting. The documentation for convert(_:to:) says it "performs the coordinate conversion as if the scene is presented inside the view." Keep an SKView that you never add to a window, give it the bounds of your MTKView, and convert the event location through it:

final class GameView: MTKView {
    var spriteRenderer: SKRenderer?

    // Never added to a window. It only converts points.
    private let converter = SKView(frame: .zero)

    override func mouseDown(with event: NSEvent) {
        guard let scene = spriteRenderer?.scene else { return }
        converter.frame = bounds
        let viewPoint = convert(event.locationInWindow, from: nil)
        let scenePoint = converter.convert(viewPoint, to: scene)
        for node in scene.nodes(at: scenePoint) {
            print(node.name ?? "unnamed", scenePoint)
        }
    }
}

Use bounds for the converter's frame rather than drawableSize. The view point from the event and bounds are both in points. The drawableSize documentation says its default value "is derived from the current view's size, in native pixels."

I tested this on macOS 27.0.1 by rendering marker sprites with SKRenderer at a backing scale factor of 2. Then I converted each sprite's rendered location with convert(_:to:) and compared the result with the sprite's position in the scene. The results matched exactly for the fill, aspectFill, and aspectFit scale modes, with two anchor points, three view aspect ratios, and an SKCameraNode. The nodes(at:) method found the expected sprite at every converted point. You can run the same check in your scene with the known-points technique from TN3124: Debugging coordinate space issues. Put a sprite at a known scene position, click it, and confirm that the converted point matches.

Converting Coordinate Spaces suggests location(in:) on NSEvent for mouse events in macOS, but it doesn't do this conversion for a scene that SKRenderer draws. The location(in:) documentation describes the node parameter as "A node that is a descendant of a scene presented in the window that received the mouse event." In my tests, it returned the point in view coordinates unchanged.