Tiled grayscale rendering to drop the storeBwBuffer peak largest
contiguous free block (the value that actually drives OOM on the C3)
from ~114 KB to ~82-90 KB.
This renders each grayscale plane band-by-band into a small (~8 KB)
scratch and
streams each band straight to controller RAM (community-sdk
writeGrayscalePlaneStrip), leaving the BW framebuffer intact. No save,
no
restore; controller RAM is re-synced for the next differential turn
directly
from the live framebuffer.
Three writers honor the active band target so per-band re-rendering
stays cheap
and correct:
- drawPixel (text) redirects writes to the band scratch and clips to it.
- renderCharImpl skips glyphs whose physical y-extent is outside the
band before
the bitmap decode (glyphIntersectsStrip), so the per-band re-render
doesn't
pay N x glyph decode.
- DirectPixelWriter (images) writes the band scratch via getWriteTarget
instead
of the framebuffer. Without this, image pixels wrote the live BW frame
directly and cleanup re-synced that corruption, leaving thin outlines
after
navigating away from an image.
Controller specifics live in the SDK (X4 setRamArea windowing, X3 PTL);
the
reader checks supportsStripGrayscale() and is otherwise
controller-agnostic.
Measured on hardware (X4 and X3, text and images, visually correct):
- Grayscale scratch ~8 KB vs ~50 KB save; largest contiguous free block
held at
full size during grayscale instead of dropping ~25-32 KB.
- X4 text page about +25 ms/page; X3 page time is dominated by its
intrinsic
grayscale refresh, not tiling.
Depends on community-sdk #13 (the writeGrayscalePlaneStrip API). The
submodule
bump here points at that branch, so until #13 merges the submodule won't
resolve
from upstream and CI will fail there; keeping this a draft until then.
Will
rebase onto master and re-point the submodule to the merged SDK commit
once #13
lands.
Did you use AI tools to help write this code? partial
SdFat's SdSpiCard tracks SPI bus state with an unsynchronized
m_spiActive bool. When two tasks call into SdFat concurrently they can
confuse that state machine, ending with one task calling
SPIClass::endTransaction() against a paramLock the other task holds.
That trips FreeRTOS's xTaskPriorityDisinherit assert (tasks.c:5156,
pxTCB == pxCurrentTCBs[0]) and panics the system.
HalStorage already serialized every explicit method call via
storageMutex, but HalFile's destructor was `= default`, which let the
underlying SdFat FsFile destructor run close() outside any lock
(DESTRUCTOR_CLOSES_FILE=1). Any task that destructed a HalFile while
another task was mid-SD-op would race the unsynchronized state.
Move the locking discipline into HalFile::Impl::~Impl: an explicit
close() under StorageLock, then the FsFile member destructor's redundant
close() is a no-op. HalFile's special members can stay = default.
Switch storageMutex to xSemaphoreCreateRecursiveMutex so openFileForRead
and openFileForWrite can hold the lock while assigning to a HalFile&
out-param whose prior Impl needs locked teardown. Priority inheritance
still applies to recursive mutexes.
Also documented the no-bypass rule in CLAUDE.md: never call SdFat /
SdSpiCard / FsBaseFile / SDCardManager directly, never define
HAL_STORAGE_IMPL outside HalStorage.cpp.
Addresses my admittedly synthetic repro for #2047
Did you use AI tools to help write this code? partial
With both OtaUpdater and HttpDownloader on esp_http_client
(https://github.com/crosspoint-reader/crosspoint-reader/pull/2074,
https://github.com/crosspoint-reader/crosspoint-reader/pull/2075),
checkForUpdate no longer needs its own client and event handler to fetch
the release JSON. It streams the response straight into
ReleaseJsonParser through a new HttpDownloader::fetchUrl(url,
DataCallback) overload, dropping the duplicate esp_http_client setup,
the HTTP_EVENT_ON_DATA handler, and the totalBytesReceived file global.
DataCallback hands body chunks to a callback without buffering. The
naive alternative, collecting the ~32KB JSON into a std::string, aborts
under -fno-exceptions: the growing allocation collides with the TLS
session's heap mid-fetch and operator new calls abort().
The OTA install path stays on esp_https_ota (flash-write streaming),
which has no HttpDownloader equivalent.
HttpDownloader.h must precede the lwip (esp_http_client) headers in
OtaUpdater.cpp, or Arduino/SdFat macros collide with lwip.
Did you use AI tools to help write this code? partial
Based on the learnings from
https://github.com/crosspoint-reader/crosspoint-reader/pull/2074 , I
wanted to bring the same buffer savings to the rest of our HTTP Client
stack. That being said, HttpDownloader (fonts/OPDS) used the Arduino
HTTPClient.
HttpDownloader was the last consumer of the Arduino HTTPClient +
NetworkClientSecure stack. OtaUpdater already runs on esp_http_client,
so this drops the parallel HTTP/TLS implementation. It also fixes a
class of OPDS/font download failures: HTTPClient's setTimeout is uint16
and truncates, and its short per-read deadline killed slow or chunked
responses (the -11 / incomplete-data errors).
What changed:
- Rewrote fetchUrl/downloadToFile around esp_http_client with a
streaming open() -> fetch_headers() -> read() loop, manual redirect
following, and is_complete_data_received() as the completeness gate.
Body bytes go straight to the sink (OPDS parser stream, std::string, or
file), so nothing buffers the payload.
- HTTPS is now verified against the CA bundle instead of
NetworkClientSecure::setInsecure(). esp-tls is built with
CONFIG_ESP_TLS_INSECURE off, so an unverified handshake can't be set up
anyway; the model is public servers over verified https and local
servers over plain http (transport is chosen from the URL scheme).
** Self-signed https servers are no longer supported, by design. **
- timeout_ms is 60s; esp_http_client's timeout is uint32, so unlike
HTTPClient it doesn't silently truncate.
- HTTP buffers are 4096 (rx) / 1024 (tx). 4096 holds real OPDS server
headers; the GitHub release CDN sends more and logs a non-fatal
truncation warning, but the headers we read (Location, Content-Length)
come first and survive.
- Removed the now-unused UrlUtils::isHttpsUrl and a stale HTTPClient
comment in FontDownloadActivity.
Validated on device: OPDS browse and a 3.4 MB book download over
verified https, GitHub font downloads (crc-checked), redirect handling
matching curl, and slow/erroring servers surfaced correctly.
Did you use AI tools to help write this code? partial
Unify the two splash-skip signals (RTC silent-reboot flag, SD
seamless-sleep flag) into one BootResume enum driving a single switch.
Storage unchanged; behavior-preserving apart from the fix.
Holding power to sleep from a WiFi activity (Font Download, OPDS, web
server, Calibre, KOReader sync) rebooted to home instead of sleeping.
goToSleep() runs the outgoing activity's onExit(), and those activities
call silentRestart() to clear heap fragmentation, so the heap-defrag
reboot fired before deep sleep could start.
enterDeepSleep() now latches deepSleepInProgress before goToSleep();
silentRestart()/silentRestartToReader() no-op while it's set. Deep sleep
is a full chip reset on wake, so it already clears the fragmentation the
reboot existed for.
Did you use AI tools to help write this code? partial
## Summary
After a silent reboot, there was a small window where the esp32 would
listen for button presses but the full refresh would hold the event
loop. This gave a UX experience where the silent reboot had completed to
the home screen, a user taps select (at any time during the process),
and they find themselves unexpectedly in a book.
## Additional Context
This must land after
https://github.com/crosspoint-reader/community-sdk/pull/11 and will need
the submodule SHA changes included in.
---
### AI Usage
While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.
Did you use AI tools to help write this code? partially
OTA install streams the full 5.8MB image over a multi-minute TLS session
while wifi/LWIP already holds the big internal arena. Measured on
device, the arena bottomed out at ~7.7KB free with the largest
contiguous block down to ~2.4KB; for 80% of the download there wasn't
even a contiguous 8KB block. It finishes on a clean heap but tips into
OOM for anyone carrying more pre-OTA fragmentation.
Two avoidable drains, both in OtaUpdater:
- The esp_http_client RX/TX buffers were 8192/8192 on both the version
check and the install. RX only has to hold response headers (bodies
stream through the parser / OTA writer) and TX only carries our GET, so
trim both to 4096/1024. 4096 still fits the github->CDN redirect
headers; the 512 IDF default truncates them, which is why they got
oversized in the first place.
- installUpdate fired the progress callback every ~100ms perform
iteration, waking the render task on every tick. Its framebuffer work
fights the TLS session for the same arena, and epd can't really repaint
faster than a percent anyway. Throttle it to whole-percent changes.
On device, combined: floor 7.7KB -> 19KB, worstcase contiguous block
2.4KB -> 21KB, zero sub 8KB iterations across the whole download.
KOReaderSyncClient already uses small buffers; HttpDownloader is on the
Arduino HTTPClient stack with no equivalent knob, so neither changed.
Did you use AI tools to help write this code? partial, heap-exploration
assisted by Claude.
setTimeout(0) on the serial could trigger a subtle but obnoxious
underflow.
Eat a milli, save a reset button.
Did you use AI tools to help write this code? no
When the device went into deep sleep while plugged into USB, a
power-button press would occasionally not wake it. The display held its
last frame, the chip stayed in deep sleep, and recovery required an
unplug + reset + hold-power cycle. On battery the symptom never surfaced
because the power button physically re-energises the chip.
Two peripherals were holding power domains alive across the deep sleep
boundary and interfering with the configured GPIO wake on the power
button:
1. HWCDC. Once Serial is initialized, the USB Serial/JTAG peripheral
keeps its power domain configured even with TX timeout at zero and no
host draining. Tear it down with Serial.end() in
HalPowerManager::startDeepSleep, gated by ENABLE_SERIAL_LOG to match the
Serial.begin site. This hit me if I was charging off my computer.
2. WiFi. enterDeepSleep had no WiFi teardown, so sleeping from any
network-using activity left the modem domain alive. Call
WiFi.disconnect(true) + WiFi.mode(WIFI_OFF) when WiFi is active. Wake
from deep sleep is effectively a chip reset, so no WiFi state needs to
survive. While this doesn't cause higher power drain, it apparently was
causing issues where I'd occasionally have the chip hang on sleep
transition from a wifi activity.
Confirmed on device.
Did you use AI tools to help write this code? partial
EIGHT_BIT_GRAYSCALE decode of a 3-component progressive JPEG calls
JPEGDecodeMCU_P with MCU_SKIP for Cb and Cr after every Y MCU. The
existing safe-pMCU patch redirects the wild pointer to &sMCUs[0] but
leaves the DC store unguarded, so each chroma skip overwrites the
just-decoded Y DC with the chroma DC predictor. Output reads sMCUs[0],
gets the trailing Cr DC (~0), and renders an all-black image.
Add `if (iMCU >= 0)` guards to the two pMCU[0] writes (main DC store and
successive-approximation update). The pointer redirect stays as the AC
wild-pointer defense; the new guards stop the silent corruption at
sMCUs[0]. The two fixes are independent and both required.
fixes the progressive 8bit grayscale jpeg regression in 1.3.0
Did you use AI tools to help write this code? partial
On-device repro showed the cover snapshot pinning ~52KB of contiguous
heap (cloning the full 48KB framebuffer with malloc overhead). MaxAlloc
on Home was 61KB; nothing was leaving headroom for HTTPS, which needs
30-50KB contiguous for the mbedTLS handshake.
Add region-aware framebuffer helpers to GfxRenderer that translate a
logical rect through rotateCoordinates and copy only the byte range that
contains the rotated rect. HomeActivity records the tile rect it passes
to drawRecentBookCover and caches only that subregion.
Measured on device (X3, Portrait):
Idle on Home | Free 102K -> 139K | MaxAlloc 61K -> 115K
Mid-EPUB-read | Free 81K -> 134K | MaxAlloc 70K -> 115K
Cover cache | ~52K -> ~16K (per allocation)
Works in all four orientations because the bounds helper samples the
four logical corners through the existing rotation, so the cached byte
range always covers the pixels the theme could have drawn into.
Savings will vary with theme, but should be significant across all.
The "logs only flow if you unplug and replug the USB cable at the right
moment" symptom traced to two interacting problems with the ESP32-C3 USB
Serial/JTAG controller (HWCDC):
1. Serial.begin was gated on gpio.isUsbConnected(). That check sampled
USB state at one specific microsecond during boot. If USB enumeration on
the host hadn't completed by that moment (common after a reset that
auto- reconnects a moment later), Serial was never initialized and
stayed dead until the next boot where the timing happened to win.
2. HWCDC writes block for up to the configured TX timeout (default 250
ms) when the host has the port open but isn't actively draining — a
state the macOS USB CDC stack enters intermittently after reconnect. The
firmware then appears to hang on logging until a USB unplug+replug
cycles the peripheral and flushes the TX FIFO.
Fix: move the Serial init to the very top of setup() with a 250 ms stall
before Serial.begin (lets the USB peripheral power-on and host
enumeration complete on cold boot), and call logSerial.setTxTimeoutMs(0)
so writes drop bytes harmlessly when the host is slow instead of
stalling the firmware. Both warm reboot and cold power-on now produce
logs immediately.
Did you use AI tools to help write this code? partial
PR #1908 silent-restarts on exit from any wifi-using activity to defuse
LWIP/mbedTLS heap fragmentation, but two of the wifi-using paths slipped
through that audit:
KOReaderAuthActivity (Settings -> KOReader sync -> Authenticate)
OtaUpdateActivity (Settings -> Check for update, back-out paths)
Both used WiFi.disconnect + WiFi.mode(WIFI_OFF) on exit and returned
control to Settings, leaving ~50KB of contiguous heap stranded for the
rest of the session.
Mirror the FontDownloadActivity pattern: if WiFi was activated,
disconnect and silentRestart. OTA's success path is unchanged:
SHUTTING_DOWN already calls plain ESP.restart() so the new firmware
boots normally; only the cancel/fail/no-update back-out paths now go
through silentRestart().
Did you use AI tools to help write this code? partial
WiFi/LWIP/netif teardown scatters long-lived allocations across the
heap, leaving ~50KB of contiguous space unrecoverable without a reboot.
Reboot the SoC on exit from any wifi-using activity to guarantee a clean
heap. An RTC_NOINIT flag survives the reboot and tells setup() to skip
the boot splash and route the user back where they came from:
- File transfer / Calibre / OPDS / Font download -> home
- KOReader sync -> currently-open EPUB
Activities check WiFi.getMode() before rebooting, so backing out of the
network mode menu without joining doesn't trigger a cycle. KOSync also
esp_wifi_stop()s after the sync result so the radio is off while the
user reads it; full teardown happens at the reboot.
## Additional Context
The silent reboot skips the booting splash screen - it visibly looks
like a screen refresh. This does cause a disconnection/reconnection blip
for developers actively pulling logs over serial, but `pio device
monitor` and the like successfully reconnect and feed in the early boot
serial.
as an example:
```
[256676] [DBG] [ACT] Exiting activity: KOReaderSync
[256706] [DBG] [MAIN] Silent restart (target=reader)
ESP-ROM:esp32c3-api1-20210207
Build:Feb 7 2021
rst:0xc (RTC_SW_CPU_RST),boot:0xf (SPI_FAST_FLASH_BOOT)
Saved PC:0x403872bc
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fcd72a0,len:0x990
load:0x403cbf10,len:0xac8
load:0x403ce710,len:0x4d28
entry 0x403cbf10
[22] [INF] [MAIN] Hardware detect: X4
[29] [SD] SD card detected
[43] [DBG] [CPS] Settings loaded from file
[58] [DBG] [KRS] Loaded KOReader credentials for user: jeremydk
[69] [DBG] [OPS] Loaded 1 OPDS servers from file
[69] [DBG] [UI] Using Lyra theme
[70] [DBG] [MAIN] Starting CrossPoint version 1.2.0-dev-detached-bde75787
...
[203] [DBG] [ACT] Entering activity: Reader
[211] [DBG] [EBP] Loading ePub: /Halting State - Charles Stross.epub
[221] [DBG] [BMC] Loaded cache data: 51 spine, 41 TOC entries
[246] [DBG] [CSS] Loaded 41 rules from cache
[247] [DBG] [EBP] Loaded ePub: /Halting State - Charles Stross.epub
```
---
### AI Usage
While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.
Did you use AI tools to help write this code? _PARTIALLY_
## Summary
* **What is the goal of this PR?**
I was seeing hangs during File Transfer. Troubleshooting via serial
showed it had to do with the SHUTTING_DOWN state. Strengthened the WiFi
State Machine a bit and added self-healing. Additionally, I found it
obnoxious to need to keep referring to serial for my wifi strength, so a
dBm meter is added opposing the SSID when on the File Transfer page.
* **What changes are included?**
The dBm meter was at risk of causing rapid screen updates (if, say, we
hovered right around a threshold), so I've implemented a basic
hysteresis around this. RISING/FALLING would be more canonical variable
names, but are reserved in the framework.
## Additional Context
My X4 has a terrible antenna, apparently, and I was running into this
failure condition pretty regularly. Wifi "bars" were chosen as a
relatively pan-cultural glyph rather than relying on localization.
Tested on hardware at -83 dBm under sustained EPUB upload (60 books, ~30
MB). Transient losses up to ~14s now ride through; pre-fix the same
losses required a power-cycle after as little as a 2s blip.
---
### AI Usage
While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.
Did you use AI tools to help write this code? Nope. Commits are hand
written. Claude was used to build a local test harness for validation
only during iteration.
Jeremy Klein