Building a Joystick Controller using Compose Multiplatform

Ready to play and learn with Compose Multiplatform? Let’s discover the Gesture APIs to craft our very own joystick.
This hands-on session is perfect for developers keen on improving their Compose skills across platforms. You'll get to mix a bit of trigonometry fun with the practical magic of Compose Multiplatform, applicable for both iOS and Android. Come along for an adventure in coding - it's going to be educational, practical, and most importantly, a lot of fun!

Transcript

1. 🤌 Filippo Scognamiglio 🥖 Renaud Mathieu Building a Joystick using Compose Multiplatform Android Makers by droidcon 2024 ❤
2. @renaud_mathieu @Renaud@androiddev.social renaudmathieu.com @Swordfish90 @flscogna@gmail.com swordfish90.github.io
3. Prologue
7. Introduction
8. Overview Touch and input in Compose Pointer Input Focus Interactions Jetpack Compose
9. Pointer Input Components Gesture Modifiers Gesture Recognizers Pointer Events
10. Jolanda Verhoef
11. Understand gestures pointerInput fun Modifier.pointerInput( key1: Any?, block: suspend PointerInputScope.() -> Unit ): Modifier = this then SuspendPointerInputElement( key1 = key1, pointerInputHandler = block )
12. Understand gestures detectDragGestures suspend fun PointerInputScope.detectDragGestures( onDragStart: (Offset) -> Unit = { }, onDragEnd: () -> Unit = { }, onDragCancel: () -> Unit = { }, onDrag: (change: PointerInputChange, dragAmount: Offset) -> Unit ): Unit
13. Part 1. The bad idea 😢
17. Joystick NavKey, NavState @Composable pointerInput(x,y)
18. Joystick NavKey, NavState @Composable pointerInput(x,y)
19. Joystick NavKey, NavState @Composable pointerInput(x,y)
20. Joystick NavKey, NavState @Composable pointerInput(x,y) enum class NavState { RELEASED, PRESSED } enum class NavKey { UNKNOWN, ARROW_LEFT, ARROW_UP, ARROW_RIGHT, ARROW_DOWN, ARROW_UPPER_LEFT, ARROW_LOWER_LEFT, ARROW_UPPER_RIGHT, ARROW_LOWER_RIGHT }
21. Joystick @Composable @Composable fun JoyStick( modifier: Modifier = Modifier, size: Dp = 200.dp, dotSize: Dp = 110.dp, onValueChanged: (NavKey, NavState) -> Unit )
25. UP UPPER_RIGHT DOWN RIGHT LOWER_RIGHT LEFT UPPER_LEFT LOWER_LEFT
26. UP UPPER_RIGHT DOWN RIGHT LOWER_RIGHT LEFT UPPER_LEFT LOWER_LEFT
27. UP UPPER_RIGHT DOWN RIGHT LOWER_RIGHT LEFT UPPER_LEFT LOWER_LEFT
28. UP UPPER_RIGHT DOWN RIGHT LOWER_RIGHT LEFT UPPER_LEFT LOWER_LEFT
29. UP UPPER_RIGHT DOWN RIGHT LOWER_RIGHT LEFT UPPER_LEFT LOWER_LEFT
31. Part 2. The Better Idea 🤔
32. 🧰 Tools A. Calculate the angle B. Find the direction from the angle C. Apply the offset to the dot
33. x,y 0,0 Calculate the angle
34. x,y 0,0 ϑ Calculate the angle
35. x,y 0,0 ϑ A B C Calculate the angle
36. x,y 0,0 ϑ A B C tan(ϑ) = BC / AC tan(ϑ) = y / x ϑ = atan(y / x) Calculate the angle
37. theta = getTheta(x, y) private fun getTheta(x: Float, y: Float) = when { x < 0 -& y -= 0 -> (PI).toFloat() + atan(y / x) x < 0 -& y < 0 -> -(PI).toFloat() + atan(y / x) else -> atan(y / x) }
38. Find the direction from the angle
39. Find the direction from the angle
40. Find the direction from the angle private fun getDirectionFromAngle(theta: Float): NavKey = when { theta -= 0 -& theta -= PI.div(6) -> { NavKey.ARROW_RIGHT } theta -= PI.div(6) -& theta -= PI.div(3) -> { NavKey.ARROW_LOWER_RIGHT } …
41. Calculate the offset of the dot Calculate the radius then the offset • Polar to Cartesian
42. Calculate the offset of the dot Calculate the radius then the offset • Polar to Cartesian private fun polarToCartesian( radius: Float, theta: Float ): Pair<Float, Float> = Pair(radius * cos(theta), radius * sin(theta))
43. Box( modifier = Modifier .paint( painter = painterResource("joystick_background.xml"), alignment = Alignment.Center ) ) { Image( painter = painterResource("joystick_dot.xml"), contentDescription = "JoyStickDot" ) }
44. val localDensity = LocalDensity.current val centerX = with(localDensity) { ((size - dotSize) / 2).toPx() } val centerY = with(localDensity) { ((size - dotSize) / 2).toPx() } Image( painter = painterResource("joystick_dot.xml"), contentDescription = "JoyStickDot", modifier = Modifier .offset { IntOffset( centerX.roundToInt(), centerY.roundToInt() ) } .size(dotSize) )
45. val localDensity = LocalDensity.current val centerX = with(localDensity) { ((size - dotSize) / 2).toPx() } val centerY = with(localDensity) { ((size - dotSize) / 2).toPx() } Image( painter = painterResource("joystick_dot.xml"), contentDescription = "JoyStickDot", modifier = Modifier .offset { IntOffset( centerX.roundToInt(), centerY.roundToInt() ) } .size(dotSize) )
46. var navKey by remember { mutableStateOf(NavKey.UNKNOWN) } var navState by remember { mutableStateOf(NavState.RELEASED) }
47. .pointerInput(Unit) { detectDragGestures( onDragStart = { navState = NavState.PRESSED }, onDragEnd = { }, onDrag = { pointerInputChange: PointerInputChange, offset: Offset -> } ) }
48. .pointerInput(Unit) { detectDragGestures( onDrag = { pointerInputChange: PointerInputChange, offset: Offset -> if (navState -= NavState.PRESSED) { -/ Nothing is pressed return@detectDragGestures } pointerInputChange.consume() val x = offsetX + offset.x - centerX val y = offsetY + offset.y - centerY theta = getTheta(x, y) val fourDimensionDirection: NavKey = getFourDimensionFromAngle(theta) if (navKey -= fourDimensionDirection) {
49. theta = getTheta(x, y) val fourDimensionDirection: NavKey = getFourDimensionFromAngle(theta) if (navKey -= fourDimensionDirection) { if (navKey -= NavKey.UNKNOWN) { onValueChanged(navKey, NavState.RELEASED) } navKey = fourDimensionDirection -/ vibrate() if (navKey -= NavKey.UNKNOWN) { onValueChanged(navKey, NavState.PRESSED) } } } ) }
50. The dot does not move 😶
51. radius = sqrt((x.pow(2)) + (y.pow(2))) offsetX += offset.x offsetY += offset.y val t = polarToCartesian(radius, theta) positionX = t.first positionY = t.second
53. val maxRadius = with(localDensity) { 30.dp.toPx() } if (radius > maxRadius) { polarToCartesian(maxRadius, theta) } else { polarToCartesian(radius, theta) }.apply { positionX = first positionY = second }
54. Image( painter = painterResource("joystick_dot.xml"), contentDescription = "JoyStickDot", modifier = Modifier .offset { IntOffset( (positionX + centerX).roundToInt(), (positionY + centerY).roundToInt() ) }
55. Drag End ?
56. onDragEnd = { -/ Reset everybody offsetX = centerX offsetY = centerY radius = 0f theta = 0f positionX = 0f positionY = 0f -/ Release everybody navState = NavState.RELEASED if (navKey -= NavKey.UNKNOWN) { releaseAllNavKeys() navKey = NavKey.UNKNOWN } }
58. Part 3. The Best Idea 🤩
59. Handling multiple controls Expected touch behaviour change depending on the different controls
60. Single gamepad state data class InputState( internal val digitalKeys: PersistentSet<Int> = persistentSetOf(), internal val directions: PersistentMap<Int, Offset> = persistentMapOf(), ) { fun setDigitalKey(keyId: Int, value: Boolean): InputState { … } fun getDigitalKey(digitalId: Int): Boolean { return digitalKeys.contains(digitalId) } fun setDirection(directionId: Int, offset: Offset): InputState { … } fun getDirection(directionId: Int, default: Offset = Offset.Unspecified): Offset { return directions.getOrElse(directionId) { default } } }
61. Single input handling • All inputs are received a single compose function • Each Control compose function is associated to a Handler • The handler receives the pointer event if: ◦ It’s tracking the pointer ◦ The pointer is currently in its trigger area • Each handler updates the state and may request to track a pointer id data class Pointer(val pointerId: Long, val position: Offset) data class Result(val inputState: InputState, val dragGestureStart: Pointer? = null) fun handle(pointers: List<Pointer>, inputState: InputState, dragGestureStart: Pointer?): Result
62. Finding the right Handler val gamepadPosition: MutableState<Offset> val trackedPointers = scope.getTrackedIds() val handlersAssociations: Map<Handler?, List<Pointer-> = event.changes .asSequence() .filter { it.pressed } .map { Pointer(it.id.value, it.position + gamepadPosition.value) } .groupBy { pointer /> if (pointer.pointerId in trackedPointers) { scope.getHandlerTracking(pointer.pointerId) } else { scope.getHandlerAtPosition(pointer.position) } }
63. Updating the state scope.inputState.value = scope.getAllHandlers().fold(scope.inputState.value) { state, handler /> val pointers = handlersAssociations.getOrElse(handler) { emptyList() } val relativePointers = pointers .map { Pointer(it.pointerId, it.position.relativeTo(handler.rect)) } val (updatedState, updatedTracked) = handler.handle(relativePointers, state, scope.getStartDragGestureForHandler(handler)) scope.setStartGestureForHandler(handler, updatedTracked) updatedState }
64. Writing layouts - Scope class JamPadScope { internal val inputState = mutableStateOf(InputState()) private val handlers = mutableMapOf<String, Handler>() --. internal fun registerHandler(handler: Handler) { --. } }
65. Writing Layouts - Controls fun JamPadScope.ControlAnalog(--.){ BoxWithConstraints( modifier = modifier.onGloballyPositioned { registerHandler(AnalogHandler(id, it.boundsInRoot())) }, ) { ... } }
66. Joystick Refined
67. Analog Handling override fun handle(pointers: List<Pointer>, inputState: InputState, startDragGesture: Pointer?): Result { val currentDragGesture = pointers.firstOrNull { it.pointerId -= startDragGesture-.pointerId } return when { pointers.isEmpty() -> { Result(inputState.setDirection(id, Offset.Unspecified), null) } startDragGesture -= null -& currentDragGesture -= null -> { val deltaPosition = (currentDragGesture.position - startDragGesture.position) val offsetValue = deltaPosition.coerceIn(Offset(-1f, -1f), Offset(1f, 1f)) Result(inputState.setDirection(id, offsetValue), startDragGesture) } else -> { val firstPointer = pointers.first() Result(inputState.setDirection(id, Offset.Zero), firstPointer) } } }
68. The Analog Control @Composable fun JamPadScope.ControlAnalog( modifier: Modifier = Modifier, id: Int, background: @Composable () -> Unit = { DefaultControlBackground() }, foreground: @Composable (Boolean) -> Unit = { DefaultButtonForeground(pressed = it, scale = 1f) }, ) { val position = inputState.value.getDirection(id, Offset.Zero) BoxWithConstraints( modifier = modifier .aspectRatio(1f) .onGloballyPositioned { registerHandler(AnalogPointerHandler(id, it.boundsInRoot())) }, contentAlignment = Alignment.Center, ) { Box(modifier = Modifier.fillMaxSize(0.75f)) { background() } Box(modifier = Modifier .fillMaxSize(0.50f) .offset(maxWidth * position.x * 0.25f, maxHeight * position.y * 0.25f), ) { foreground(inputState.value.getDirection(id) -= Offset.Unspecified) } } }
69. Cross Handling enum class State(val position: Offset) { UP(Offset(0f, 1f)), DOWN(Offset(0f, -1f)), LEFT(Offset(-1f, 0f)), RIGHT(Offset(1f, 0f)), UP_LEFT(Offset(-1f, 1f)), UP_RIGHT(Offset(1f, 1f)), DOWN_LEFT(Offset(-1f, -1f)), DOWN_RIGHT(Offset(1f, -1f)), }
70. override fun handle( pointers: List<Pointer>, inputState: InputState, startDragGesture: Pointer?, ): Result { val currentDragGesture = pointers.firstOrNull { it.pointerId -= startDragGesture-.pointerId } return when { pointers.isEmpty() -> { Result( inputState.setDirection(id, Offset.Unspecified), null ) } currentDragGesture -= null -> { Result( inputState.setDirection(id, findCloserState(currentDragGesture)), startDragGesture ) } else -> { val firstPointer = pointers.first() Result( inputState.setDirection(id, findCloserState(firstPointer)), firstPointer ) } } }
71. The Cross Control @Composable fun JamPadScope.ControlCross( modifier: Modifier = Modifier, id: Int, background: @Composable () -> Unit = { DefaultControlBackground() }, foreground: @Composable (Offset) -> Unit = { DefaultCrossForeground(direction = it) }, ) { Box( modifier = modifier .aspectRatio(1f) .onGloballyPositioned { registerHandler(CrossPointerHandler(id, it.boundsInRoot())) }, ) { background() foreground(inputState.value.getDirection(id)) } }
72. Face buttons Face buttons have many faces:
73. Face buttons We need to allow customization Depending on the number of buttons we lay them down in a circle. For each pointer in the circle we compute the closest button.
74. Face buttons fun computeSizeOfItemsOnCircumference(itemsCount: Int): Float { val angle = sin(Constants.PI / maxOf(itemsCount, 2)) return (angle / (1 + angle)) }
75. Composite buttons Some games require you to press two Face Buttons and the same time. We simulate this with composite buttons.
76. Haptics Feeling a button press is very important because the brain needs to register the input happens. We can use Kotlin Multiplatform actual expect mechanism to implement the Android and iOS variants. When the input state changes we try can emit a press or release feedback.
77. Haptics enum class HapticEffect { PRESS, RELEASE, } interface HapticGenerator { fun generate(type: HapticEffect) } @Composable expect fun rememberHapticGenerator(): HapticGenerator
78. iOS Haptics object IosHapticGenerator : HapticGenerator { private val impactFeedbackGenerator = UIImpactFeedbackGenerator() override fun generate(type: HapticEffect) { when (type) { HapticEffect.PRESS -> impactFeedbackGenerator.impactOccurredWithIntensity(0.5) HapticEffect.RELEASE -> impactFeedbackGenerator.impactOccurredWithIntensity(0.3) } } }
79. Android Haptics class AndroidHapticGenerator(applicationContext: Context) : HapticGenerator { … override fun generate(type: HapticEffect) { val effect = when (type) { HapticEffect.PRESS -> strongEffect HapticEffect.RELEASE -> weakEffect } vibrator.vibrate(effect) }
80. A minimal gamepad JamPad( modifier = Modifier.fillMaxSize().aspectRatio(2f), onInputStateUpdated = { … } ) { // Inside JamPad scope… Row(modifier = Modifier.fillMaxSize()) { ControlCross( modifier = Modifier.weight(1f), id = 0 ) ControlFaceButtons( modifier = Modifier.weight(1f), ids = listOf(1, 2, 3) ) } }
81. Introducing JamPadCompose A Virtual Gamepad Library for Kotlin Multiplatform https://github.com/piepacker/JamPadCompose • The library is still in its early stages of development • Contributions and feedback are highly welcome to improve the library and expand its capabilities.
83. Epilogue
86. Thank you 🙏 Special thanks to @renaud_mathieu @Renaud@androiddev.social renaudmathieu.com @fil @fil fil.com