Adds residentBytes() and cssRuleCount() methods to track EPUB memory usage including parsed CSS. Enhances memory audit logging with EPUB heap usage, max allocatable block size, and CSS rule count. Implements one-shot heap block map dump on first settled audit to identify fragmentation and long-lived allocations.
Implements reportMemory() methods across SdCardFont, SdCardFontManager, and Section classes to track resident heap usage. Adds heap audit logging in EpubReaderActivity to attribute memory to fonts, framebuffer, section data, and other components for measurement-driven optimization.
Preload glyphs for the next page during idle time to reduce page-turn latency. After a 400ms debounce period, scan the next page in FCM scan mode (no pixels drawn) to cache missing glyphs from SD card, avoiding ~100ms of SD reads during the actual page turn. Prewarm is deferred when rendering is active, heap is low, or during rapid page-flipping.
Buffer JSON file list entries into ~1.4KB batches before sending to avoid one TCP segment per file. This eliminates client delayed ACK delays that were causing large directory listings to take tens of seconds. Falls back to per-entry sends if buffer allocation fails.
Explicitly stop the BLE stack before bringing up WiFi in CrossPointWebServerActivity, FontDownloadActivity, and OtaUpdateActivity. On ESP32-C3, the shared radio and heap between BLE and WiFi stacks causes WiFi to be permanently starved if initialized while NimBLE's ~50KB is still resident, as the WiFi driver sizes its RX/TX buffer pools at init time.
Previously the status bar showed 'BT connecting' until the next page turn. Now monitors connection state in loop() and redraws the status bar immediately when connection completes or disconnects, restoring the chapter/book title. The connecting message now takes over the entire title slot instead of prepending to it.
Introduces non-blocking display refresh methods that allow CPU work (like grayscale rendering) to overlap with the e-ink panel's refresh time. The async path starts the waveform and returns immediately, with the caller responsible for waiting via waitRefreshComplete(). Falls back to blocking refresh when fadingFix is enabled or the panel lacks deferral support.
Changes all sleep screen display refreshes and boot screens from
FULL_REFRESH to HALF_REFRESH to match OEM firmware behavior. The stock
firmware uses a single-pass 0xD7 waveform for sleep screens, not the
multi-flash GC waveform (0xF7) that FULL_REFRESH triggers. For grayscale
mode, HALF_REFRESH is required because the gray nudge LUT is calibrated
against the pixel charge state left by the single-pass waveform; using
FULL_REFRESH causes blotchy noise in gray areas. Also adds X3 wb_gc
passive byte (the one-byte 0x54 to 0x00, matching stock's passive WB
table in the gc nudge bank): Every X3 grayscale render, reader AA,
covers, and the sleep-cover nudge, gets it.
Display a Bluetooth icon in the status bar when BLE is connected. Also optimize memory management by lending the framebuffer instead of immediately tearing down BLE when heap is low, allowing BLE to remain active during rendering.
Temporarily release framebuffer memory during section pagination to reduce heap pressure. The framebuffer can be released before memory-intensive operations and restored afterward, allowing BLE stack startup threshold to be lowered from 80KB to 70KB. Implements RAII-style FrameBufferBuildLoan class to ensure proper cleanup and automatic restart on restoration failure.
Add missingInclude suppression to prevent false positives on fresh CI checkouts where include paths are unresolved. Add inline suppression for constVariableReference warning where cppcheck cannot detect mutations without full include path resolution.
Reduce BLE stack memory usage from ~68KB to ~52KB by disabling unused NimBLE roles (peripheral, broadcaster), limiting connections to 1, and reducing buffer counts for ACL, HCI events, and ATT entries. Remove unused cloud components (esp_insights, esp_rainmaker) that require unavailable server certificates. These changes prevent stack collision with the render shed, eliminating restart flaps on the reader device.
Replace per-page buffer free+realloc pattern with capacity tracking that only reallocates when needed size exceeds current capacity. This prevents heap fragmentation from non-coalescing holes that occurred when each page's freed block rarely fit the next page's allocation. After a few page turns, capacities converge on the book's maximum and page turns stop touching the allocator entirely.
Prevent OOM aborts by shedding BLE stack when free heap drops below 24KB before rendering. Lower BLE start threshold from 100KB to 80KB to match steady-state heap levels (~84KB). Reserve vector capacity upfront in page parsing to avoid reallocation crashes on fragmented heaps. Gate background section builds on heap availability.
Introduces a global pause flag to temporarily block BLE stack auto-start during large memory allocations (e.g., in EPUB rendering). Also adds auto-restart logic for BLE scanning when device list is empty, and extensive debug logging for BLE scan lifecycle troubleshooting.
This PR moves us from the xteink openx4 SDK to the freeink sdk from
https://freeink.org. Out of the box there are NO changes needed in the
firmware to support this swap, it all magically works as is. However as
we support more than just the x3/x4 devices, this sdk allows us to pass
env vars into the build commands to include support for other devices.
As support for new hardware such as touch screens and bluetooth are
added the xteink builds decide at compile time if the libraries are used
or not. For example right now the freeinkui and icons libraries are in
the platform.io file but as they are not used anywhere, they won't be
included in the final build. Once the touch branch and sd themes branch
are merged in this sdk is required for them to function correctly. All
the docs for freeink are available at freeink.org/docs. x4/x3 is a
single binary build unlike other devices that will build unique binaries
for each device. Eventually we will want to remove a lot of the manual
isx3 type stuff from our firmware and go through the boardsupport api
the sdk provides as it will generalize everything into one common system
that any device can support. The upcoming touch branch does a lot of
this for us but this initial PR is JUST to get the sdk swapped over
without any code changes to show seamless integration without any
regressions.
Force display resync on X3 when HALF refresh is requested to clear prior content before rendering. Add grayscale preconditioning for X3's UC81xx controller to even out single-pixel dithering artifacts that appear as speckle with its turbo BW waveform.
Introduces a virtual method allowing activities to indicate whether they need the BLE stack to remain active. This enables selective teardown of Bluetooth to free heap memory, with the Bluetooth settings screen being able to override this for pairing and scanning functionality.
When building an EPUB section fails with Bluetooth enabled, temporarily stop the BLE stack to free up memory (~16KB), retry the build, then restart BLE. This works around memory fragmentation caused by the NimBLE stack that prevents allocating the large contiguous buffer needed for inflate/deflate operations. The recovery only runs once per uncached chapter since chapters are cached afterwards.
Integrate BLE keyboard host functionality for page-turner remotes. Adds device pairing, button mapping, preset configurations for Free2/Free3 remotes, and persistent storage of mappings. Migrates from open-x4-sdk to freeink-sdk submodule which includes the BleKeyboardHost library. Implements CPU frequency locking during BLE operations to prevent watchdog timeouts.
(#2334)
Introduces a new display method that prepares the framebuffer as a base
frame for grayscale overlays. On X3 panels, this uses the OEM
differential base waveform (AA-pre-BW) without forcing a resync. Other
panels fall back to normal display with configurable refresh mode.
Dependent upon matching SDK commit to work
Pulls in community-sdk PR #9, which clears inGrayscaleMode inside
cleanupGrayscaleBuffers() after the restored BW frame is written back
into RED RAM. Without this, the next BW page turn would still see the
flag set and trigger a redundant grayscaleRevert() refresh, producing
visible ghosting on the X4 with text anti-aliasing enabled.
Regression introduced by SDK commit 0a8ada2 (factory LUT grayscale
support), which removed a redundant inGrayscaleMode guard in
grayscaleRevert() and so caused the cleanup to actually run for the
first time.
Bypassing rules to avoid this going stale and all nightly builds being broken for x4 users
Document the Xteink Unlocker tool requirement for third-party purchased
xteink units that ship with USB flashing locked. Include warnings about
bricking risks when flashing unsupported firmwares (e.g. Papyrix) on
locked devices, as they may permanently lock the device with no recovery
path.
Adds royalty.dev as a funding platform for contributors by adding custom
funding link and badge to README
A maintainer would need to click the "Are you the maintainer?" button on
https://app.royalty.dev/crosspoint-reader/crosspoint-reader to install
the github app which pulls the PRs in and ranks them.
(Disclosure, Royalty is a product I've recently built and am using for
my own OSS and currently gating custom font builds and nightlys on
crosspointreader.com)
This page currently can collect donations but without the integration it
wont notify contributors and it wont rescore fresh PRs.
Switch KOReader sync progress mapping from chapter matching to
XPath-based mapping.
- resolves KOReader positions using real XHTML ancestry paths
- supports paragraph-based upload mapping with text offsets
where needed
- passes the current paragraph index into sync so uploads map
back to KOReader more accurately
No HTTP client changes are included. No reader-state or resume-flow
changes are included.
---------
Co-authored-by: jpirnay <jens@pirnay.com>
Opening XTC files with a high page count (e.g. *The Magic Mountain* at
4,187 pages) causes an immediate `abort()` crash and reboot loop. The
device becomes unusable until the book is removed from the SD card.
**Crash log:**
```
abort() was called at 0x4214a5fb on core 0
```
### Root cause
During `XtcParser::open()`, the parser calls
`m_pageTable.resize(pageCount)` to load the entire page table into RAM.
Each `PageInfo` entry is 16 bytes, so:
- 4,187 pages x 16 bytes = **66,992 bytes (~65KB)** as a single
contiguous heap allocation
On the ESP32-C3 with ~380KB total RAM (no PSRAM), this allocation fails
after firmware, fonts, and the activity system are already loaded.
Because the firmware is compiled with `-fno-exceptions`, the failed
`new` inside `std::vector::resize()` calls `abort()` instead of
throwing.
This affects any XTC file with roughly 3,000+ pages, depending on heap
state at the time of loading.
## Solution
Replace the bulk page table allocation with on-demand reads from the SD
card. Instead of loading all page table entries into a vector at file
open, we now:
1. Read only the **first** page table entry at open time (to get default
page dimensions)
2. Read a **single** 16-byte entry from the SD card each time a page is
loaded
This reduces page table memory usage from `pageCount * 16` bytes to
**zero bytes**, regardless of how many pages the file contains.
### Changes
| File | What changed |
|------|-------------|
| `XtcParser.h` | Removed `std::vector<PageInfo> m_pageTable`. Added
`readPageTableEntry()` for on-demand reads. |
| `XtcParser.cpp` | Replaced `readPageTable()` with
`readFirstPageInfo()`. Updated `getPageInfo()`, `loadPage()`, and
`loadPageStreaming()` to seek and read individual entries from the file.
|
## Trade-offs
### Performance
Each page turn now requires one additional SD card seek + 16-byte read
to look up the page table entry before reading the page data itself.
- SD card sequential read latency: ~0.1-0.5ms for a 16-byte read
- E-ink full display refresh: ~1,000-2,000ms
I personally can't see any performance difference while reading and the
trade off of not boot looping seems to make this well worth it.
### Memory
| Metric | Before | After |
|--------|--------|-------|
| Page table RAM (4,187 pages) | ~65KB | 0 bytes |
| Page table RAM (1,000 pages) | ~16KB | 0 bytes |
| Page table RAM (max 65,535 pages) | ~1MB (impossible) | 0 bytes |
Adds a PlatformIO pre-build script to patch JPEGDEC library. When
decoding progressive JPEGs with AC coefficients, MCU_SKIP (-8) causes
array index 0xFFFFF8, creating a wild pointer ~33MB past the sMCUs
array. The patch redirects pMCU to sMCUs[0] when MCU_SKIP is active,
preventing store-access faults while maintaining correct behavior for
JPEG_SCALE_EIGHTH decoding. Devices with larger framebuffers (like the
x3) (792×528 = 52,272 bytes vs 800×480 = 48,000 bytes) have less free
heap, shifting the allocation and changing where the wild pointer lands.
Commit 8628297 guarded the DC coefficient write (pMCU[0]) with if (iMCU
>= 0), which prevents crashes for progressive JPEGs whose first scan is
DC-only (iScanEnd == 0). However, if the first scan includes AC
coefficients (iScanEnd > 0), the AC decode loop still writes through the
wild pointer and crashes.
Replace hardcoded DISPLAY_WIDTH, DISPLAY_HEIGHT, and DISPLAY_WIDTH_BYTES
constants with runtime values from display object to support multiple
device models (X3 and X4) with different screen dimensions.
## Summary
Adds Xteink X3 hardware support to CrossPoint Reader. The X3 uses the
same SSD1677 e-ink controller as the X4 but with a different panel
(792x528 vs 800x480), different button layout, and an I2C fuel gauge
(BQ27220) instead of ADC-based battery reading.
All X3-specific behavior is gated by runtime device detection — X4
behavior is unchanged.
Depends on community-sdk X3 support: open-x4-epaper/community-sdk#19
(merged).
## Changes
### HAL Layer
**HalGPIO** (`lib/hal/HalGPIO.cpp/.h`)
- I2C-based device fingerprinting at boot: probes for BQ27220 fuel
gauge, DS3231 RTC, and QMI8658 IMU to distinguish X3 from X4
- Detection result cached in NVS for fast subsequent boots
- Exposes `deviceIsX3()` / `deviceIsX4()` helpers used throughout the
codebase
- X3 button mapping (7 GPIOs vs X4's layout)
- USB connection detection and wake classification for X3
**HalDisplay** (`lib/hal/HalDisplay.cpp/.h`)
- Calls `einkDisplay.setDisplayX3()` before init when X3 is detected
- Requests display resync after power button / flash wake events
- Runtime display dimension accessors (`getDisplayWidth()`,
`getDisplayHeight()`, `getBufferSize()`)
- Exposed as global `display` instance for use by image converters
**HalPowerManager** (`lib/hal/HalPowerManager.cpp/.h`)
- X3 battery reading via I2C fuel gauge (BQ27220 at 0x55, SOC register)
- X3 power button uses GPIO hold for deep sleep
### Display & Rendering
**GfxRenderer** (`lib/GfxRenderer/GfxRenderer.cpp/.h`)
- Buffer size and display dimensions are now runtime values (not
compile-time constants) to support both panel sizes
- X3 anti-aliasing tuning: only the darker grayscale level is applied to
avoid washed-out text on the X3 panel. X4 retains both levels via
`deviceIsX4()` gate
**Image Converters** (`lib/JpegToBmpConverter`, `lib/PngToBmpConverter`)
- Cover image prescale target uses runtime display dimensions from HAL
instead of hardcoded 800x480
### UI Themes
**BaseTheme / LyraTheme** (`src/components/themes/`)
- X3 button position mapping for the different physical layout
- Adjusted UI element positioning for 792x528 viewport
### Boot & Init
**main.cpp**
- X3 hardware detection logging
- Adjusted init sequence for X3 (no `HalSystem::begin()` dependency on
X3 path)
**HomeActivity**
- Uses runtime `renderer.getBufferSize()` instead of static
`GfxRenderer::getBufferSize()`
FYI I did not add support for the gyro page turner. That can be it's own
PR.
This partially fixes#769 but is dependent upon PR #827 being merged
along side this @daveallie. I removed my PNG conversion code after
finding out that PR was already created. Without this PR though that
book in #769 will still fail to load because of how it's stored in the
file
---------
Co-authored-by: Dave Allie <dave@daveallie.com>
I had an epub that just showed a blank cover and wouldnt work for the
sleep screen either, turns out it was an epub3 and I guess we didn't
support that. Super simple fix here
Our esp32 consistently dropped the last few packets of the TCP transfer
in the old implementation. Only about 1/5 transfers would complete. I've
refactored that entire system into an actual Calibre Device Plugin that
basically uses the exact same system as the web server's file transfer
protocol. I kept them separate so that we don't muddy up the existing
file transfer stuff even if it's basically the same at the end of the
day I didn't want to limit our ability to change it later.
I've also added basic auth to OPDS and renamed that feature to OPDS
Browser to just disassociate it from Calibre.
---------
Co-authored-by: Arthur Tazhitdinov <lisnake@gmail.com>
Co-authored-by: Dave Allie <dave@daveallie.com>
## Summary
- Adds KOReader progress sync integration, allowing CrossPoint to sync
reading positions with other
KOReader-compatible devices
- Stores credentials securely with XOR obfuscation
- Uses KOReader's partial MD5 document hashing for cross-device book
matching
- Syncs position via percentage with estimated XPath for compatibility
# Features
- Settings: KOReader Username, Password, and Authenticate options
- Sync from chapters menu: "Sync Progress" option appears when
credentials are configured
- Bidirectional sync: Can apply remote progress or upload local progress
---------
Co-authored-by: Dave Allie <dave@daveallie.com>
## Summary
Adds support for browsing and downloading books from a Calibre-web
server via OPDS.
How it works
1. Configure server URL in Settings → Calibre Web URL (e.g.,
https://myserver.com:port I use Cloudflare tunnel to make my server
accessible anywhere fwiw)
2. "Calibre Library" will now show on the the home screen
3. Browse the catalog - navigate through categories like "By Newest",
"By Author", "By Series", etc.
4. Download books - select a book and press Confirm to download the EPUB
to your device
Navigation
- Up/Down - Move through entries
- Confirm - Open folder or download book
- Back - Go to parent catalog, or exit to home if at root
- Navigation entries show with > prefix, books show title and author
- Button hints update dynamically ("Open" for folders, "Download" for
books)
Technical details
- Fetches OPDS catalog from {server_url}/opds
- Parses both navigation feeds (catalog links) and acquisition feeds
(downloadable books)
- Maintains navigation history stack for back navigation
- Handles absolute paths in OPDS links correctly (e.g.,
/books/opds/navcatalog/...)
- Downloads EPUBs directly to the SD card root
Note
The server URL should be typed to include https:// if the server
requires it - HTTP→HTTPS redirects may cause SSL errors on ESP32.
## Additional Context
* I also changed the home titles to use uppercase for each word and
added a setting to change the size of the side margins
---------
Co-authored-by: Dave Allie <dave@daveallie.com>
Justin Mitchell