Changes the container backend strategy to use SD catalog by default for all EPUBs. On first open or stale index, builds the catalog once by borrowing the framebuffer (which can be released since e-ink holds the loading popup image). Subsequent opens use the fast path with compact resident tables. Falls back to legacy in-RAM open if catalog build fails due to IoError or OOM.
Implement cover extraction for books stored in catalog.fibc omnibus format to avoid loading large (~400KB) book arenas into RAM. Uses catalog's pre-resolved cover entries with minimal memory footprint (~42KB resident). Includes arena management for catalog opening and ZIP entry extraction with configurable scratch buffers.
Implements a two-tier system for handling EPUB books: small books use in-RAM metadata (probe ladder with 48KB/72KB/96KB rungs), while large containers (e.g. webnovel omnibuses with 1500+ spine items) automatically fall back to SD-backed BookCatalog with persistent index. The catalog fast path reuses existing catalog.fibc files to skip probing on subsequent opens. Adds framebuffer borrowing for larger probe rungs and incremental chapter building for giant chapters.
Introduce methods to temporarily release the 48KB framebuffer during memory-intensive chapter builds, allowing the memory to be reused. Add font metadata query methods (hasGlyphMeta, ensureAdvance) that check glyph existence and fetch advance metrics without loading full bitmap data, avoiding expensive SD reads during layout measurement.
Replace fixed-size arena buffers with two-pass allocation strategy. First pass parses book into temporary full-size arena to determine actual memory footprint (5-40 KB vs 48 KB cap), then reallocates exact-sized buffers. This reduces peak memory usage by avoiding simultaneous allocation of all arenas with chapter-build scratch buffer (~100 KB) on constrained heap (~140 KB free). Stylesheet is also compacted to actual size, and all open-time transients are freed before per-chapter arenas are allocated.
Refactors book cover and metadata handling out of Epub class into a dedicated BookCoverUtils utility. Moves FootnoteEntry struct into reader activity directory. Simplifies cache clearing to use direct storage operations instead of Epub class.
Implements bidirectional conversion between KOReader XPath positions and FreeInkBook character offsets. Uses streaming SAX parsing to match chapter text accounting with ChapterLayout, enabling position sync without requiring DOM or pagination. Handles element ancestry tracking, text offset calculation, and spine index extraction.
Implements layout-independent bookmarks using character offsets within chapters instead of percentage-based positions. This allows bookmarks to restore to the exact position regardless of font size or orientation changes. Adds TOC navigation helpers, book progress calculation based on spine item sizes, and a NONE BidiBaseDir option for pre-reordered text. Legacy bookmarks without character offsets fall back to percentage-based positioning.
This PR moves us from the xteink openx4 SDK to the freeink sdk from
https://freeink.org. Out of the box there are NO changes needed in the
firmware to support this swap, it all magically works as is. However as
we support more than just the x3/x4 devices, this sdk allows us to pass
env vars into the build commands to include support for other devices.
As support for new hardware such as touch screens and bluetooth are
added the xteink builds decide at compile time if the libraries are used
or not. For example right now the freeinkui and icons libraries are in
the platform.io file but as they are not used anywhere, they won't be
included in the final build. Once the touch branch and sd themes branch
are merged in this sdk is required for them to function correctly. All
the docs for freeink are available at freeink.org/docs. x4/x3 is a
single binary build unlike other devices that will build unique binaries
for each device. Eventually we will want to remove a lot of the manual
isx3 type stuff from our firmware and go through the boardsupport api
the sdk provides as it will generalize everything into one common system
that any device can support. The upcoming touch branch does a lot of
this for us but this initial PR is JUST to get the sdk swapped over
without any code changes to show seamless integration without any
regressions.
Force display resync on X3 when HALF refresh is requested to clear prior content before rendering. Add grayscale preconditioning for X3's UC81xx controller to even out single-pixel dithering artifacts that appear as speckle with its turbo BW waveform.
(#2334)
Introduces a new display method that prepares the framebuffer as a base
frame for grayscale overlays. On X3 panels, this uses the OEM
differential base waveform (AA-pre-BW) without forcing a resync. Other
panels fall back to normal display with configurable refresh mode.
Dependent upon matching SDK commit to work
Pulls in community-sdk PR #9, which clears inGrayscaleMode inside
cleanupGrayscaleBuffers() after the restored BW frame is written back
into RED RAM. Without this, the next BW page turn would still see the
flag set and trigger a redundant grayscaleRevert() refresh, producing
visible ghosting on the X4 with text anti-aliasing enabled.
Regression introduced by SDK commit 0a8ada2 (factory LUT grayscale
support), which removed a redundant inGrayscaleMode guard in
grayscaleRevert() and so caused the cleanup to actually run for the
first time.
Bypassing rules to avoid this going stale and all nightly builds being broken for x4 users
Document the Xteink Unlocker tool requirement for third-party purchased
xteink units that ship with USB flashing locked. Include warnings about
bricking risks when flashing unsupported firmwares (e.g. Papyrix) on
locked devices, as they may permanently lock the device with no recovery
path.
Adds royalty.dev as a funding platform for contributors by adding custom
funding link and badge to README
A maintainer would need to click the "Are you the maintainer?" button on
https://app.royalty.dev/crosspoint-reader/crosspoint-reader to install
the github app which pulls the PRs in and ranks them.
(Disclosure, Royalty is a product I've recently built and am using for
my own OSS and currently gating custom font builds and nightlys on
crosspointreader.com)
This page currently can collect donations but without the integration it
wont notify contributors and it wont rescore fresh PRs.
Switch KOReader sync progress mapping from chapter matching to
XPath-based mapping.
- resolves KOReader positions using real XHTML ancestry paths
- supports paragraph-based upload mapping with text offsets
where needed
- passes the current paragraph index into sync so uploads map
back to KOReader more accurately
No HTTP client changes are included. No reader-state or resume-flow
changes are included.
---------
Co-authored-by: jpirnay <jens@pirnay.com>
Opening XTC files with a high page count (e.g. *The Magic Mountain* at
4,187 pages) causes an immediate `abort()` crash and reboot loop. The
device becomes unusable until the book is removed from the SD card.
**Crash log:**
```
abort() was called at 0x4214a5fb on core 0
```
### Root cause
During `XtcParser::open()`, the parser calls
`m_pageTable.resize(pageCount)` to load the entire page table into RAM.
Each `PageInfo` entry is 16 bytes, so:
- 4,187 pages x 16 bytes = **66,992 bytes (~65KB)** as a single
contiguous heap allocation
On the ESP32-C3 with ~380KB total RAM (no PSRAM), this allocation fails
after firmware, fonts, and the activity system are already loaded.
Because the firmware is compiled with `-fno-exceptions`, the failed
`new` inside `std::vector::resize()` calls `abort()` instead of
throwing.
This affects any XTC file with roughly 3,000+ pages, depending on heap
state at the time of loading.
## Solution
Replace the bulk page table allocation with on-demand reads from the SD
card. Instead of loading all page table entries into a vector at file
open, we now:
1. Read only the **first** page table entry at open time (to get default
page dimensions)
2. Read a **single** 16-byte entry from the SD card each time a page is
loaded
This reduces page table memory usage from `pageCount * 16` bytes to
**zero bytes**, regardless of how many pages the file contains.
### Changes
| File | What changed |
|------|-------------|
| `XtcParser.h` | Removed `std::vector<PageInfo> m_pageTable`. Added
`readPageTableEntry()` for on-demand reads. |
| `XtcParser.cpp` | Replaced `readPageTable()` with
`readFirstPageInfo()`. Updated `getPageInfo()`, `loadPage()`, and
`loadPageStreaming()` to seek and read individual entries from the file.
|
## Trade-offs
### Performance
Each page turn now requires one additional SD card seek + 16-byte read
to look up the page table entry before reading the page data itself.
- SD card sequential read latency: ~0.1-0.5ms for a 16-byte read
- E-ink full display refresh: ~1,000-2,000ms
I personally can't see any performance difference while reading and the
trade off of not boot looping seems to make this well worth it.
### Memory
| Metric | Before | After |
|--------|--------|-------|
| Page table RAM (4,187 pages) | ~65KB | 0 bytes |
| Page table RAM (1,000 pages) | ~16KB | 0 bytes |
| Page table RAM (max 65,535 pages) | ~1MB (impossible) | 0 bytes |
Adds a PlatformIO pre-build script to patch JPEGDEC library. When
decoding progressive JPEGs with AC coefficients, MCU_SKIP (-8) causes
array index 0xFFFFF8, creating a wild pointer ~33MB past the sMCUs
array. The patch redirects pMCU to sMCUs[0] when MCU_SKIP is active,
preventing store-access faults while maintaining correct behavior for
JPEG_SCALE_EIGHTH decoding. Devices with larger framebuffers (like the
x3) (792×528 = 52,272 bytes vs 800×480 = 48,000 bytes) have less free
heap, shifting the allocation and changing where the wild pointer lands.
Commit 8628297 guarded the DC coefficient write (pMCU[0]) with if (iMCU
>= 0), which prevents crashes for progressive JPEGs whose first scan is
DC-only (iScanEnd == 0). However, if the first scan includes AC
coefficients (iScanEnd > 0), the AC decode loop still writes through the
wild pointer and crashes.
Replace hardcoded DISPLAY_WIDTH, DISPLAY_HEIGHT, and DISPLAY_WIDTH_BYTES
constants with runtime values from display object to support multiple
device models (X3 and X4) with different screen dimensions.
## Summary
Adds Xteink X3 hardware support to CrossPoint Reader. The X3 uses the
same SSD1677 e-ink controller as the X4 but with a different panel
(792x528 vs 800x480), different button layout, and an I2C fuel gauge
(BQ27220) instead of ADC-based battery reading.
All X3-specific behavior is gated by runtime device detection — X4
behavior is unchanged.
Depends on community-sdk X3 support: open-x4-epaper/community-sdk#19
(merged).
## Changes
### HAL Layer
**HalGPIO** (`lib/hal/HalGPIO.cpp/.h`)
- I2C-based device fingerprinting at boot: probes for BQ27220 fuel
gauge, DS3231 RTC, and QMI8658 IMU to distinguish X3 from X4
- Detection result cached in NVS for fast subsequent boots
- Exposes `deviceIsX3()` / `deviceIsX4()` helpers used throughout the
codebase
- X3 button mapping (7 GPIOs vs X4's layout)
- USB connection detection and wake classification for X3
**HalDisplay** (`lib/hal/HalDisplay.cpp/.h`)
- Calls `einkDisplay.setDisplayX3()` before init when X3 is detected
- Requests display resync after power button / flash wake events
- Runtime display dimension accessors (`getDisplayWidth()`,
`getDisplayHeight()`, `getBufferSize()`)
- Exposed as global `display` instance for use by image converters
**HalPowerManager** (`lib/hal/HalPowerManager.cpp/.h`)
- X3 battery reading via I2C fuel gauge (BQ27220 at 0x55, SOC register)
- X3 power button uses GPIO hold for deep sleep
### Display & Rendering
**GfxRenderer** (`lib/GfxRenderer/GfxRenderer.cpp/.h`)
- Buffer size and display dimensions are now runtime values (not
compile-time constants) to support both panel sizes
- X3 anti-aliasing tuning: only the darker grayscale level is applied to
avoid washed-out text on the X3 panel. X4 retains both levels via
`deviceIsX4()` gate
**Image Converters** (`lib/JpegToBmpConverter`, `lib/PngToBmpConverter`)
- Cover image prescale target uses runtime display dimensions from HAL
instead of hardcoded 800x480
### UI Themes
**BaseTheme / LyraTheme** (`src/components/themes/`)
- X3 button position mapping for the different physical layout
- Adjusted UI element positioning for 792x528 viewport
### Boot & Init
**main.cpp**
- X3 hardware detection logging
- Adjusted init sequence for X3 (no `HalSystem::begin()` dependency on
X3 path)
**HomeActivity**
- Uses runtime `renderer.getBufferSize()` instead of static
`GfxRenderer::getBufferSize()`
FYI I did not add support for the gyro page turner. That can be it's own
PR.
This partially fixes#769 but is dependent upon PR #827 being merged
along side this @daveallie. I removed my PNG conversion code after
finding out that PR was already created. Without this PR though that
book in #769 will still fail to load because of how it's stored in the
file
---------
Co-authored-by: Dave Allie <dave@daveallie.com>
I had an epub that just showed a blank cover and wouldnt work for the
sleep screen either, turns out it was an epub3 and I guess we didn't
support that. Super simple fix here
Our esp32 consistently dropped the last few packets of the TCP transfer
in the old implementation. Only about 1/5 transfers would complete. I've
refactored that entire system into an actual Calibre Device Plugin that
basically uses the exact same system as the web server's file transfer
protocol. I kept them separate so that we don't muddy up the existing
file transfer stuff even if it's basically the same at the end of the
day I didn't want to limit our ability to change it later.
I've also added basic auth to OPDS and renamed that feature to OPDS
Browser to just disassociate it from Calibre.
---------
Co-authored-by: Arthur Tazhitdinov <lisnake@gmail.com>
Co-authored-by: Dave Allie <dave@daveallie.com>
## Summary
- Adds KOReader progress sync integration, allowing CrossPoint to sync
reading positions with other
KOReader-compatible devices
- Stores credentials securely with XOR obfuscation
- Uses KOReader's partial MD5 document hashing for cross-device book
matching
- Syncs position via percentage with estimated XPath for compatibility
# Features
- Settings: KOReader Username, Password, and Authenticate options
- Sync from chapters menu: "Sync Progress" option appears when
credentials are configured
- Bidirectional sync: Can apply remote progress or upload local progress
---------
Co-authored-by: Dave Allie <dave@daveallie.com>
## Summary
Adds support for browsing and downloading books from a Calibre-web
server via OPDS.
How it works
1. Configure server URL in Settings → Calibre Web URL (e.g.,
https://myserver.com:port I use Cloudflare tunnel to make my server
accessible anywhere fwiw)
2. "Calibre Library" will now show on the the home screen
3. Browse the catalog - navigate through categories like "By Newest",
"By Author", "By Series", etc.
4. Download books - select a book and press Confirm to download the EPUB
to your device
Navigation
- Up/Down - Move through entries
- Confirm - Open folder or download book
- Back - Go to parent catalog, or exit to home if at root
- Navigation entries show with > prefix, books show title and author
- Button hints update dynamically ("Open" for folders, "Download" for
books)
Technical details
- Fetches OPDS catalog from {server_url}/opds
- Parses both navigation feeds (catalog links) and acquisition feeds
(downloadable books)
- Maintains navigation history stack for back navigation
- Handles absolute paths in OPDS links correctly (e.g.,
/books/opds/navcatalog/...)
- Downloads EPUBs directly to the SD card root
Note
The server URL should be typed to include https:// if the server
requires it - HTTP→HTTPS redirects may cause SSL errors on ESP32.
## Additional Context
* I also changed the home titles to use uppercase for each word and
added a setting to change the size of the side margins
---------
Co-authored-by: Dave Allie <dave@daveallie.com>
Justin Mitchell