222 lines
12 KiB
Markdown
222 lines
12 KiB
Markdown
# Touch and UI Development
|
|
|
|
CrossPoint now runs on touch devices (Seeed Sticky, M5Paper, LilyGo T5) alongside the button-only Xteink X3/X4. Every screen must work with both input styles. There are two supported ways to get there:
|
|
|
|
1. **New screens: build them with FreeInkUI components.** Touch hit-testing, tap highlighting, long-press, and button focus navigation come with the component; you never hand-roll coordinate math.
|
|
2. **Existing screens and in-flight features: use the MappedInputManager touch bridge.** A small set of helpers adds tap/hold/swipe support to hand-rolled rendering without restructuring the activity.
|
|
|
|
If you are starting a new activity, use FreeInkUI. If you have a feature branch with a hand-rolled screen already working on buttons, use the bridge; do not rewrite mid-flight.
|
|
|
|
---
|
|
|
|
## Path 1: New screens with FreeInkUI
|
|
|
|
FreeInkUI (`freeink-sdk/libs/ui/FreeInkUI`, namespace `freeink::ui`) is an immediate-mode component library. The core idea:
|
|
|
|
- While a component renders, it registers its tappable areas ("hit rects") into the frame's interaction buffer via `Frame::hit(rect, action, value, inputMask, state)`.
|
|
- Each loop you build an `InputSnapshot` from the input manager and route it against that buffer. If a tap (or mapped button press) lands in a registered rect, the routed `ActionEvent` tells you which action fired and with what value.
|
|
- Components query `frame.stateFor(action, value)` while painting, so touch-down highlight and focus states render correctly without any per-activity code.
|
|
|
|
You get touch, hold highlighting, long-press, minimum touch-target sizing, and orientation-aware coordinates for free. The same action IDs fire from physical buttons, so one code path serves both input styles.
|
|
|
|
### Component inventory
|
|
|
|
All under `freeink-sdk/libs/ui/FreeInkUI/include/components/`:
|
|
|
|
| Category | Components |
|
|
|---|---|
|
|
| Controls | `button`, `checkbox`, `slider`, `progress-bar`, `header` |
|
|
| Lists | `list` (virtualized), `table`, `dropdown`, `radio-group`, `setting-row`, `toggle-row`, `stepper-row` |
|
|
| Keyboard | `keyboard` (QWERTY/AZERTY/QWERTZ/ES layouts), `key-grid` |
|
|
| Overlays | `popup`, `option-dialog`, `context-menu`, `message-panel`, `toast` |
|
|
| Bars | `status-bar`, `tab-bar`, `reader-chrome`, `battery-indicator`, `gesture-bar`, `tap-zones` |
|
|
| Media | `book-card`, `cover-grid`, `cover-carousel`, `metric-card` |
|
|
| Text | `text-field`, `text-area` |
|
|
|
|
### Integration skeleton
|
|
|
|
The in-tree reference is [`src/activities/util/KeyboardEntryActivity.cpp`](../../src/activities/util/KeyboardEntryActivity.cpp); it drives the FreeInkUI keyboard component inside a normal Activity. The shape:
|
|
|
|
```cpp
|
|
#include <FreeInkUIGfxRenderer.h>
|
|
namespace fui = freeink::ui;
|
|
|
|
// Render step: draw the component and (re)register its hit rects.
|
|
fui::GfxRendererTarget target(renderer); // adapts GfxRenderer to FreeInkUI's DrawTarget
|
|
target.setFont(fui::GfxRendererTarget::FONT_BODY, UI_12_FONT_ID);
|
|
const fui::DeviceContext device = target.deviceContext(); // orientation, safe area, touch sizing
|
|
|
|
fui::Frame<48> frame(target, device, fui::InputSnapshot{}, interactions); // 48 = max hit rects
|
|
fui::KeyboardProps props;
|
|
props.layout = ¤tLayout();
|
|
props.keyAction = ACTION_KEY; // one action id; the key is the event value
|
|
props.inputMask = fui::InputTouch | fui::InputLongPress;
|
|
fui::keyboard(frame, kbRect, props); // draws AND registers hit rects
|
|
```
|
|
|
|
```cpp
|
|
// Input step (each loop): feed touch state, act on routed events.
|
|
int tx, ty, tapX, tapY, hx, hy;
|
|
const bool pressedDown = mappedInput.wasScreenTouchDown(tx, ty);
|
|
const bool tapped = mappedInput.wasScreenTapped(tapX, tapY);
|
|
const bool inContact = mappedInput.isScreenTouchHeld(hx, hy);
|
|
|
|
const auto result = touchRouter.update(interactions, pressedDown, tx, ty,
|
|
tapped, tapX, tapY, inContact, millis());
|
|
if (result.event) {
|
|
activateValue(result.event.value, result.event.longPress);
|
|
requestUpdate();
|
|
} else if (result.activeChanged) {
|
|
requestUpdate(); // repaint touch-down highlight
|
|
}
|
|
```
|
|
|
|
Notes:
|
|
|
|
- `Frame<N>` is templated on the max number of hit rects; the buffer is stack/static, no heap.
|
|
- `fui::TouchHoldRouter` (see `components/keyboard/keyboard.h`) synthesizes long-press while the finger is still down and swallows the eventual release. Use it whenever a component distinguishes tap from long-press.
|
|
- For simpler screens without long-press, `FreeInkApp` / `Screen<N>` (`FreeInkApp.h`) is the ergonomic layer: it owns the interaction buffer, offers a top-to-bottom builder (`takeTop`, footer helpers, etc.), and its `render(input)` call routes the snapshot and dispatches registered action callbacks in one step. `snapshotFrom(input, device)` in `FreeInkUIInputManager.h` builds the `InputSnapshot` with orientation-aware touch mapping.
|
|
- Everything stays allocation-free and works on the button-only devices unchanged: physical buttons route through the same interaction table via each hit rect's `inputMask`.
|
|
|
|
---
|
|
|
|
## Path 2: Adding touch to existing hand-rolled screens
|
|
|
|
For activities that draw their own rows, menus, and buttons, `MappedInputManager` ([src/MappedInputManager.h](../../src/MappedInputManager.h)) exposes bridge helpers. They all return **logical screen coordinates** (orientation already applied via `GfxRenderer::tapToLogical`); never touch raw normalized panel coordinates or the SDK `InputManager` directly.
|
|
|
|
| Helper | Use for |
|
|
|---|---|
|
|
| `wasScreenTapped(x, y)` | A completed tap (press + release), with logical coords |
|
|
| `wasScreenTouchDown(x, y)` | Touch-down (held > 90 ms, not yet released): draw selection highlight |
|
|
| `isScreenTouchHeld(x, y)` | Live contact position while the finger is down (drag tracking) |
|
|
| `wasTapInRect(x, y, w, h)` | One-off hit test on a rectangle (a single button, a banner) |
|
|
| `wasListItemTapped(index, count, selected, listTop, listHeight, hasSubtitle)` | Taps on a standard UITheme list; does the row/paging math for you |
|
|
| `wasListItemTouchedDown(...)` | Same geometry, touch-down phase (highlight before activate) |
|
|
| `rowTouch(row, top, rowStep, rowCount, xStart, xEnd, rowHeight)` | Any custom band of equal-height rows; returns `RowTouch::None/Down/Tap` |
|
|
| `colTouch(col, left, colStep, colCount, yStart, yEnd, colWidth)` | Horizontal button bands (dialogs, prompts) |
|
|
| `wasSwipe()` | Raw swipe direction if you need one beyond the global gestures |
|
|
| `hasTouch()` | True when the device has a touch panel (rarely needed; helpers simply never fire without one) |
|
|
|
|
### Pattern A: standard themed list
|
|
|
|
One call per loop; UITheme owns the row geometry ([EpubReaderBookmarksActivity.cpp:123](../../src/activities/reader/EpubReaderBookmarksActivity.cpp)):
|
|
|
|
```cpp
|
|
int tapped = -1;
|
|
if (mappedInput.wasListItemTapped(tapped, bookmarks.size(), selectorIndex, listY, listHeight, true)) {
|
|
selectorIndex = tapped;
|
|
openBookmark();
|
|
return;
|
|
}
|
|
```
|
|
|
|
### Pattern B: custom rows with hold highlight
|
|
|
|
For non-theme row layouts, use `rowTouch` and distinguish `Down` (highlight) from `Tap` (activate), as in [EpubReaderFootnotesActivity.cpp](../../src/activities/reader/EpubReaderFootnotesActivity.cpp):
|
|
|
|
```cpp
|
|
int row = -1;
|
|
const auto touch = mappedInput.rowTouch(row, listTop, lineHeight, visibleCount,
|
|
contentX, contentX + contentWidth);
|
|
if (touch != MappedInputManager::RowTouch::None) {
|
|
const int touched = scrollOffset + row;
|
|
if (touch == MappedInputManager::RowTouch::Down) {
|
|
if (selectedIndex != touched) { selectedIndex = touched; requestUpdate(); }
|
|
} else { // RowTouch::Tap
|
|
selectedIndex = touched;
|
|
activateSelection();
|
|
}
|
|
return;
|
|
}
|
|
```
|
|
|
|
The `Down` state exists because e-ink repaints are slow: highlight on touch-down gives immediate feedback, activation happens on release.
|
|
|
|
### Global gestures: do not reimplement these
|
|
|
|
Three gestures are handled once, for every screen. Activities must not add their own edge-swipe handling:
|
|
|
|
| Gesture | Trigger | Where it is handled |
|
|
|---|---|---|
|
|
| Back | Right-swipe starting in the left 25% of the screen | Folded into `Button::Back`, so the existing `wasPressed(Button::Back)` in your activity already fires |
|
|
| Home | Up-swipe starting in the bottom 14% | `ActivityManager::loop()`; pops to Home (activities can override via `handleHomeGesture()`) |
|
|
| Menu | Down-swipe starting in the top 14% | Activities that have a menu check `wasMenuGesture()` themselves (the reader does this) |
|
|
|
|
Because the back gesture arrives as `Button::Back`, most button-era activities gain back-swipe support with zero changes. That is the bar to aim for: bridge helpers should make touch an additive layer over the button flow, not a second input state machine.
|
|
|
|
### Bridge rules
|
|
|
|
- Handle touch in `loop()` next to the existing button handling, one helper call per interaction zone, and `return` after consuming an event (mirrors the button pattern).
|
|
- Never call the SDK `InputManager` or read GPIO directly; the HAL rule from the main guide applies to touch too.
|
|
- Coordinates from the helpers are logical and orientation-correct on all four rotations; test at least Portrait and one Landscape mode before PR.
|
|
- Nothing to clean up in `onExit()`; the helpers are stateless from the activity's point of view.
|
|
|
|
---
|
|
|
|
## Building and testing on non-Xteink devices
|
|
|
|
Each MCU family is its own binary: X3/X4 are ESP32-C3, Sticky and LilyGo T5 are ESP32-S3, M5Paper v1.1 is a classic ESP32. The Sticky env ships in `platformio.ini` (`pio run -e sticky`). Envs for other devices go in **`platformio.local.ini`**, a gitignored file that PlatformIO merges over `platformio.ini` (see `extra_configs`). Create it next to `platformio.ini`; personal envs, ports, and debug flags live there and never get committed.
|
|
|
|
Both envs below extend the repo's `[base]`, so they build against the `freeink-sdk` submodule with all the normal deps and scripts.
|
|
|
|
### M5Paper v1.1 (classic ESP32, IT8951 panel)
|
|
|
|
```ini
|
|
[env:m5paper_v11]
|
|
extends = base
|
|
board = esp32dev
|
|
board_build.mcu = esp32
|
|
board_build.flash_mode = qio
|
|
; CP2104 UART bridge: 921600 drops out on macOS after the stub baud switch
|
|
upload_speed = 460800
|
|
build_unflags =
|
|
${base.build_unflags}
|
|
; classic ESP32 has UART serial, not USB CDC; Logging.h keys off these
|
|
-DARDUINO_USB_MODE=1
|
|
-DARDUINO_USB_CDC_ON_BOOT=1
|
|
build_flags =
|
|
${base.build_flags}
|
|
-DFREEINK_DEVICE_M5PAPER=1
|
|
; the 63KB 540x960 framebuffer lives in PSRAM (FREEINK_FB_PSRAM auto-on)
|
|
-DBOARD_HAS_PSRAM
|
|
-DCROSSPOINT_VERSION=\"${crosspoint.version}-m5paper\"
|
|
-DENABLE_SERIAL_LOG
|
|
-DLOG_LEVEL=2
|
|
; touch-first device: hide front-button hint labels
|
|
-DCROSSPOINT_SHOW_BUTTON_HINTS=0
|
|
; archive-scan-order workaround: without these a full relink drops Wire's i2c symbols
|
|
-Wl,-u,i2cInit
|
|
-Wl,-u,i2cSlaveInit
|
|
```
|
|
|
|
### LilyGo T5 S3 (ESP32-S3, controller-less panel via LovyanGFX)
|
|
|
|
```ini
|
|
[env:lilygo_t5s3]
|
|
extends = base
|
|
board = esp32-s3-devkitc1-n16r8
|
|
board_build.mcu = esp32s3
|
|
build_flags =
|
|
${base.build_flags}
|
|
-DFREEINK_DEVICE_LILYGO=1
|
|
; board injects the parallel-bus pins + PMIC power hooks (BoardT5S3)
|
|
-DFREEINK_LGFX_EPD_CONFIG=lilygoT5S3LgfxConfig
|
|
-DCROSSPOINT_VERSION=\"${crosspoint.version}-lilygo\"
|
|
-DENABLE_SERIAL_LOG
|
|
-DLOG_LEVEL=2
|
|
-DCROSSPOINT_SHOW_BUTTON_HINTS=0
|
|
lib_deps =
|
|
${base.lib_deps}
|
|
; LgfxEpdConfig for the T5 S3 (pins, PCA9535/TPS65185 power sequence)
|
|
BoardT5S3=symlink://freeink-sdk/libs/hardware/BoardT5S3
|
|
; LovyanGFX Panel_EPD drives the controller-less ED047TC1 panel
|
|
m5stack/M5GFX @ 0.2.20
|
|
```
|
|
|
|
Then `pio run -e m5paper_v11 -t upload` (or `-e lilygo_t5s3`). Gotchas worth knowing:
|
|
|
|
- **Flash mode matters.** The M5Paper is `qio`; the X4-family standalone envs need `dio`. A wrong flash-mode header boots into a `partition 0 invalid magic number 0xffff` loop even though esptool verified the write.
|
|
- **One `FREEINK_DEVICE_*` flag per env** selects the board profile (pins, panel, touch controller) from the SDK's `BoardConfig`. See `freeink-sdk/platformio.sample.ini` for reference envs of every supported device.
|
|
- **Serial logs:** `[base]` does not enable logging; without `-DENABLE_SERIAL_LOG` a non-default env prints nothing.
|
|
- No touch hardware on your desk? The X4 build still exercises the same code paths through buttons; touch-specific behavior (tap zones, gestures) needs a real device.
|