feat: Add touch coordinate mapping and RTOS task yielding (#2481)
Co-authored-by: Julia Nguyen <julia@uxj.io>
This commit is contained in:
Justin Mitchell
co-authored by
Julia Nguyen
Julia Nguyen
parent
c9188a7347
commit
f42fab1c66
@@ -0,0 +1,221 @@
|
||||
# Touch and UI Development
|
||||
|
||||
CrossPoint now runs on touch devices (Seeed Sticky, M5Paper, LilyGo T5) alongside the button-only Xteink X3/X4. Every screen must work with both input styles. There are two supported ways to get there:
|
||||
|
||||
1. **New screens: build them with FreeInkUI components.** Touch hit-testing, tap highlighting, long-press, and button focus navigation come with the component; you never hand-roll coordinate math.
|
||||
2. **Existing screens and in-flight features: use the MappedInputManager touch bridge.** A small set of helpers adds tap/hold/swipe support to hand-rolled rendering without restructuring the activity.
|
||||
|
||||
If you are starting a new activity, use FreeInkUI. If you have a feature branch with a hand-rolled screen already working on buttons, use the bridge; do not rewrite mid-flight.
|
||||
|
||||
---
|
||||
|
||||
## Path 1: New screens with FreeInkUI
|
||||
|
||||
FreeInkUI (`freeink-sdk/libs/ui/FreeInkUI`, namespace `freeink::ui`) is an immediate-mode component library. The core idea:
|
||||
|
||||
- While a component renders, it registers its tappable areas ("hit rects") into the frame's interaction buffer via `Frame::hit(rect, action, value, inputMask, state)`.
|
||||
- Each loop you build an `InputSnapshot` from the input manager and route it against that buffer. If a tap (or mapped button press) lands in a registered rect, the routed `ActionEvent` tells you which action fired and with what value.
|
||||
- Components query `frame.stateFor(action, value)` while painting, so touch-down highlight and focus states render correctly without any per-activity code.
|
||||
|
||||
You get touch, hold highlighting, long-press, minimum touch-target sizing, and orientation-aware coordinates for free. The same action IDs fire from physical buttons, so one code path serves both input styles.
|
||||
|
||||
### Component inventory
|
||||
|
||||
All under `freeink-sdk/libs/ui/FreeInkUI/include/components/`:
|
||||
|
||||
| Category | Components |
|
||||
|---|---|
|
||||
| Controls | `button`, `checkbox`, `slider`, `progress-bar`, `header` |
|
||||
| Lists | `list` (virtualized), `table`, `dropdown`, `radio-group`, `setting-row`, `toggle-row`, `stepper-row` |
|
||||
| Keyboard | `keyboard` (QWERTY/AZERTY/QWERTZ/ES layouts), `key-grid` |
|
||||
| Overlays | `popup`, `option-dialog`, `context-menu`, `message-panel`, `toast` |
|
||||
| Bars | `status-bar`, `tab-bar`, `reader-chrome`, `battery-indicator`, `gesture-bar`, `tap-zones` |
|
||||
| Media | `book-card`, `cover-grid`, `cover-carousel`, `metric-card` |
|
||||
| Text | `text-field`, `text-area` |
|
||||
|
||||
### Integration skeleton
|
||||
|
||||
The in-tree reference is [`src/activities/util/KeyboardEntryActivity.cpp`](../../src/activities/util/KeyboardEntryActivity.cpp); it drives the FreeInkUI keyboard component inside a normal Activity. The shape:
|
||||
|
||||
```cpp
|
||||
#include <FreeInkUIGfxRenderer.h>
|
||||
namespace fui = freeink::ui;
|
||||
|
||||
// Render step: draw the component and (re)register its hit rects.
|
||||
fui::GfxRendererTarget target(renderer); // adapts GfxRenderer to FreeInkUI's DrawTarget
|
||||
target.setFont(fui::GfxRendererTarget::FONT_BODY, UI_12_FONT_ID);
|
||||
const fui::DeviceContext device = target.deviceContext(); // orientation, safe area, touch sizing
|
||||
|
||||
fui::Frame<48> frame(target, device, fui::InputSnapshot{}, interactions); // 48 = max hit rects
|
||||
fui::KeyboardProps props;
|
||||
props.layout = ¤tLayout();
|
||||
props.keyAction = ACTION_KEY; // one action id; the key is the event value
|
||||
props.inputMask = fui::InputTouch | fui::InputLongPress;
|
||||
fui::keyboard(frame, kbRect, props); // draws AND registers hit rects
|
||||
```
|
||||
|
||||
```cpp
|
||||
// Input step (each loop): feed touch state, act on routed events.
|
||||
int tx, ty, tapX, tapY, hx, hy;
|
||||
const bool pressedDown = mappedInput.wasScreenTouchDown(tx, ty);
|
||||
const bool tapped = mappedInput.wasScreenTapped(tapX, tapY);
|
||||
const bool inContact = mappedInput.isScreenTouchHeld(hx, hy);
|
||||
|
||||
const auto result = touchRouter.update(interactions, pressedDown, tx, ty,
|
||||
tapped, tapX, tapY, inContact, millis());
|
||||
if (result.event) {
|
||||
activateValue(result.event.value, result.event.longPress);
|
||||
requestUpdate();
|
||||
} else if (result.activeChanged) {
|
||||
requestUpdate(); // repaint touch-down highlight
|
||||
}
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `Frame<N>` is templated on the max number of hit rects; the buffer is stack/static, no heap.
|
||||
- `fui::TouchHoldRouter` (see `components/keyboard/keyboard.h`) synthesizes long-press while the finger is still down and swallows the eventual release. Use it whenever a component distinguishes tap from long-press.
|
||||
- For simpler screens without long-press, `FreeInkApp` / `Screen<N>` (`FreeInkApp.h`) is the ergonomic layer: it owns the interaction buffer, offers a top-to-bottom builder (`takeTop`, footer helpers, etc.), and its `render(input)` call routes the snapshot and dispatches registered action callbacks in one step. `snapshotFrom(input, device)` in `FreeInkUIInputManager.h` builds the `InputSnapshot` with orientation-aware touch mapping.
|
||||
- Everything stays allocation-free and works on the button-only devices unchanged: physical buttons route through the same interaction table via each hit rect's `inputMask`.
|
||||
|
||||
---
|
||||
|
||||
## Path 2: Adding touch to existing hand-rolled screens
|
||||
|
||||
For activities that draw their own rows, menus, and buttons, `MappedInputManager` ([src/MappedInputManager.h](../../src/MappedInputManager.h)) exposes bridge helpers. They all return **logical screen coordinates** (orientation already applied via `GfxRenderer::tapToLogical`); never touch raw normalized panel coordinates or the SDK `InputManager` directly.
|
||||
|
||||
| Helper | Use for |
|
||||
|---|---|
|
||||
| `wasScreenTapped(x, y)` | A completed tap (press + release), with logical coords |
|
||||
| `wasScreenTouchDown(x, y)` | Touch-down (held > 90 ms, not yet released): draw selection highlight |
|
||||
| `isScreenTouchHeld(x, y)` | Live contact position while the finger is down (drag tracking) |
|
||||
| `wasTapInRect(x, y, w, h)` | One-off hit test on a rectangle (a single button, a banner) |
|
||||
| `wasListItemTapped(index, count, selected, listTop, listHeight, hasSubtitle)` | Taps on a standard UITheme list; does the row/paging math for you |
|
||||
| `wasListItemTouchedDown(...)` | Same geometry, touch-down phase (highlight before activate) |
|
||||
| `rowTouch(row, top, rowStep, rowCount, xStart, xEnd, rowHeight)` | Any custom band of equal-height rows; returns `RowTouch::None/Down/Tap` |
|
||||
| `colTouch(col, left, colStep, colCount, yStart, yEnd, colWidth)` | Horizontal button bands (dialogs, prompts) |
|
||||
| `wasSwipe()` | Raw swipe direction if you need one beyond the global gestures |
|
||||
| `hasTouch()` | True when the device has a touch panel (rarely needed; helpers simply never fire without one) |
|
||||
|
||||
### Pattern A: standard themed list
|
||||
|
||||
One call per loop; UITheme owns the row geometry ([EpubReaderBookmarksActivity.cpp:123](../../src/activities/reader/EpubReaderBookmarksActivity.cpp)):
|
||||
|
||||
```cpp
|
||||
int tapped = -1;
|
||||
if (mappedInput.wasListItemTapped(tapped, bookmarks.size(), selectorIndex, listY, listHeight, true)) {
|
||||
selectorIndex = tapped;
|
||||
openBookmark();
|
||||
return;
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern B: custom rows with hold highlight
|
||||
|
||||
For non-theme row layouts, use `rowTouch` and distinguish `Down` (highlight) from `Tap` (activate), as in [EpubReaderFootnotesActivity.cpp](../../src/activities/reader/EpubReaderFootnotesActivity.cpp):
|
||||
|
||||
```cpp
|
||||
int row = -1;
|
||||
const auto touch = mappedInput.rowTouch(row, listTop, lineHeight, visibleCount,
|
||||
contentX, contentX + contentWidth);
|
||||
if (touch != MappedInputManager::RowTouch::None) {
|
||||
const int touched = scrollOffset + row;
|
||||
if (touch == MappedInputManager::RowTouch::Down) {
|
||||
if (selectedIndex != touched) { selectedIndex = touched; requestUpdate(); }
|
||||
} else { // RowTouch::Tap
|
||||
selectedIndex = touched;
|
||||
activateSelection();
|
||||
}
|
||||
return;
|
||||
}
|
||||
```
|
||||
|
||||
The `Down` state exists because e-ink repaints are slow: highlight on touch-down gives immediate feedback, activation happens on release.
|
||||
|
||||
### Global gestures: do not reimplement these
|
||||
|
||||
Three gestures are handled once, for every screen. Activities must not add their own edge-swipe handling:
|
||||
|
||||
| Gesture | Trigger | Where it is handled |
|
||||
|---|---|---|
|
||||
| Back | Right-swipe starting in the left 25% of the screen | Folded into `Button::Back`, so the existing `wasPressed(Button::Back)` in your activity already fires |
|
||||
| Home | Up-swipe starting in the bottom 14% | `ActivityManager::loop()`; pops to Home (activities can override via `handleHomeGesture()`) |
|
||||
| Menu | Down-swipe starting in the top 14% | Activities that have a menu check `wasMenuGesture()` themselves (the reader does this) |
|
||||
|
||||
Because the back gesture arrives as `Button::Back`, most button-era activities gain back-swipe support with zero changes. That is the bar to aim for: bridge helpers should make touch an additive layer over the button flow, not a second input state machine.
|
||||
|
||||
### Bridge rules
|
||||
|
||||
- Handle touch in `loop()` next to the existing button handling, one helper call per interaction zone, and `return` after consuming an event (mirrors the button pattern).
|
||||
- Never call the SDK `InputManager` or read GPIO directly; the HAL rule from the main guide applies to touch too.
|
||||
- Coordinates from the helpers are logical and orientation-correct on all four rotations; test at least Portrait and one Landscape mode before PR.
|
||||
- Nothing to clean up in `onExit()`; the helpers are stateless from the activity's point of view.
|
||||
|
||||
---
|
||||
|
||||
## Building and testing on non-Xteink devices
|
||||
|
||||
Each MCU family is its own binary: X3/X4 are ESP32-C3, Sticky and LilyGo T5 are ESP32-S3, M5Paper v1.1 is a classic ESP32. The Sticky env ships in `platformio.ini` (`pio run -e sticky`). Envs for other devices go in **`platformio.local.ini`**, a gitignored file that PlatformIO merges over `platformio.ini` (see `extra_configs`). Create it next to `platformio.ini`; personal envs, ports, and debug flags live there and never get committed.
|
||||
|
||||
Both envs below extend the repo's `[base]`, so they build against the `freeink-sdk` submodule with all the normal deps and scripts.
|
||||
|
||||
### M5Paper v1.1 (classic ESP32, IT8951 panel)
|
||||
|
||||
```ini
|
||||
[env:m5paper_v11]
|
||||
extends = base
|
||||
board = esp32dev
|
||||
board_build.mcu = esp32
|
||||
board_build.flash_mode = qio
|
||||
; CP2104 UART bridge: 921600 drops out on macOS after the stub baud switch
|
||||
upload_speed = 460800
|
||||
build_unflags =
|
||||
${base.build_unflags}
|
||||
; classic ESP32 has UART serial, not USB CDC; Logging.h keys off these
|
||||
-DARDUINO_USB_MODE=1
|
||||
-DARDUINO_USB_CDC_ON_BOOT=1
|
||||
build_flags =
|
||||
${base.build_flags}
|
||||
-DFREEINK_DEVICE_M5PAPER=1
|
||||
; the 63KB 540x960 framebuffer lives in PSRAM (FREEINK_FB_PSRAM auto-on)
|
||||
-DBOARD_HAS_PSRAM
|
||||
-DCROSSPOINT_VERSION=\"${crosspoint.version}-m5paper\"
|
||||
-DENABLE_SERIAL_LOG
|
||||
-DLOG_LEVEL=2
|
||||
; touch-first device: hide front-button hint labels
|
||||
-DCROSSPOINT_SHOW_BUTTON_HINTS=0
|
||||
; archive-scan-order workaround: without these a full relink drops Wire's i2c symbols
|
||||
-Wl,-u,i2cInit
|
||||
-Wl,-u,i2cSlaveInit
|
||||
```
|
||||
|
||||
### LilyGo T5 S3 (ESP32-S3, controller-less panel via LovyanGFX)
|
||||
|
||||
```ini
|
||||
[env:lilygo_t5s3]
|
||||
extends = base
|
||||
board = esp32-s3-devkitc1-n16r8
|
||||
board_build.mcu = esp32s3
|
||||
build_flags =
|
||||
${base.build_flags}
|
||||
-DFREEINK_DEVICE_LILYGO=1
|
||||
; board injects the parallel-bus pins + PMIC power hooks (BoardT5S3)
|
||||
-DFREEINK_LGFX_EPD_CONFIG=lilygoT5S3LgfxConfig
|
||||
-DCROSSPOINT_VERSION=\"${crosspoint.version}-lilygo\"
|
||||
-DENABLE_SERIAL_LOG
|
||||
-DLOG_LEVEL=2
|
||||
-DCROSSPOINT_SHOW_BUTTON_HINTS=0
|
||||
lib_deps =
|
||||
${base.lib_deps}
|
||||
; LgfxEpdConfig for the T5 S3 (pins, PCA9535/TPS65185 power sequence)
|
||||
BoardT5S3=symlink://freeink-sdk/libs/hardware/BoardT5S3
|
||||
; LovyanGFX Panel_EPD drives the controller-less ED047TC1 panel
|
||||
m5stack/M5GFX @ 0.2.20
|
||||
```
|
||||
|
||||
Then `pio run -e m5paper_v11 -t upload` (or `-e lilygo_t5s3`). Gotchas worth knowing:
|
||||
|
||||
- **Flash mode matters.** The M5Paper is `qio`; the X4-family standalone envs need `dio`. A wrong flash-mode header boots into a `partition 0 invalid magic number 0xffff` loop even though esptool verified the write.
|
||||
- **One `FREEINK_DEVICE_*` flag per env** selects the board profile (pins, panel, touch controller) from the SDK's `BoardConfig`. See `freeink-sdk/platformio.sample.ini` for reference envs of every supported device.
|
||||
- **Serial logs:** `[base]` does not enable logging; without `-DENABLE_SERIAL_LOG` a non-default env prints nothing.
|
||||
- No touch hardware on your desk? The X4 build still exercises the same code paths through buttons; touch-specific behavior (tap zones, gestures) needs a real device.
|
||||
Reference in New Issue
Block a user