74b8cac928e909e97600395b22efdb20e445e3a0
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Wylan SwetsandClaude Sonnet 4.6
|
bf894fd343 |
fix: improve KOSync bidirectional position matching accuracy (#1897)
## Summary **Goal:** Fix bidirectional KOSync position matching between CrossPoint and KOReader so that syncing in either direction lands on the correct page with character-level accuracy. **Changes included:** **Download — `toCrossPoint` (server XPath → CrossPoint page)** - **XPath ancestry mode for structured elements**: The previous `ParagraphStreamer` only tracked `<p>` elements. Replaced with a full ancestor-walking mode that correctly resolves XPaths pointing into `<li>`, `<ul>`, and other structured elements. Char offset within the target element is bounded to the matched element's content only. - **Slash-in-attribute-value corrupts depth tracking**: `processByteInTag()` treated every `/` byte as a self-closing tag marker, including `/` inside quoted attribute values (e.g. `xmlns="http://..."`, `src="Links/image.jpg"`). This drove `htmlDepth` to 0 prematurely, causing the ancestry search to exit far short of the target paragraph. Fixed with `inAttrQuote` tracking. - **Off-by-one in page formula**: `intra * totalPages` rounds up incorrectly for last-page positions. Changed to `intra * (totalPages - 1)` to map the `[0, 1]` intra fraction correctly onto the `[0, totalPages-1]` page range. Example: page 14 of 17 was returned as 15. **Upload — `toKOReader` (CrossPoint page → server XPath)** - **Off-by-one in page-to-intra formula**: Symmetric fix — `pageNumber / totalPages` changed to `pageNumber / (totalPages - 1)`, with the guard updated from `> 0` to `> 1` to avoid division by zero. - **`<li>`-based XPath generation**: When the current page starts on a list item, `findXPathForProgress` now generates `ul[N]/li[M]` XPaths rather than falling back to the preceding `<p>`. Requires the new `listItemIndex` field in `PageLutEntry` (section cache version bumped to 23). - **Text-node precision with correct `text()[N].M` format**: KOReader expects `text()[N].M` where `N` is the 1-based index of the specific text node within the element. The previous attempt generated `text().M` (no brackets), which caused KOReader to jump to the front of the book. Implements a per-element text-node index stack in `XPathProgressResolver` — parallel to the existing element path stack — that correctly tracks text node indices relative to each element. Empty text nodes from bare anchor elements (`<a id="anchor"/>`) are intentionally skipped, matching KOReader's own text node counting behavior. **Reviewer-caught bugs** - **Double `onCloseTag()` on malformed `</br/>`**: Both the `tagIsClose` path and the self-closing `/` check were firing, double-decrementing `htmlDepth`. Fixed with a `!tagIsClose` guard. - **Dangling pointer in `LOG_DBG`**: `std::to_string(*nextParagraphPage).c_str()` passed a pointer to a temporary destroyed before the variadic call. Fixed with `snprintf` into a stack `char[8]` buffer. ## Additional Context - Section cache version bumped from 22 → 23 due to the new `listItemIndex` field in `PageLutEntry`. Users upgrading will see a one-time re-render of all cached sections on first load — no data loss. - The `textNodeIndexStack` in `XPathProgressResolver` is a `std::vector<int>` that mirrors the existing `path` and `parentStates` stacks — same depth, same lifetime. No additional heap pressure beyond what was already present. - All fixes verified on device with *Gentle and Lowly* by Dane C. Ortlund (spine 21, 17 pages). Download syncs land on the correct page; upload syncs land at the correct paragraph with character-level offset. ## Test plan - [ ] Download: sync from KOReader → CrossPoint lands on correct page for `text()[N].M` XPaths - [ ] Download: ancestry correctly resolves `<li>` positions inbound from KOReader - [ ] Upload: sync from CrossPoint → KOReader lands within one page for mid-paragraph positions - [ ] Upload: sync from CrossPoint → KOReader correctly targets `<li>` elements when page starts on a list item - [ ] Upload: `text()[N].M` format XPaths do not cause KOReader to jump to front of book - [ ] Section cache version 23: delete `.crosspoint/` and verify clean re-parse with no crashes --- ### AI Usage Did you use AI tools to help write this code? **YES** — developed with Claude Code (Anthropic). --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Wylan SwetsandClaude Sonnet 4.6
|
e64155ed63 |
fix: free Epub RAM and simplify KOSync navigation via ActivityManager (#1860)
## Summary * **What is the goal of this PR?** Fix KOSync failing with "Network error" on large/complex EPUBs, and simplify the sync navigation flow by removing the callback/result pattern. * **What changes are included?** This PR combines the approaches from #1855 and #1760 into a single, cleaner solution: **Memory fix (from #1855):** - `EpubReaderActivity` pre-computes the local KOReader position and chapter name, then explicitly releases `epub` and `section` before launching `KOReaderSyncActivity`. This frees ~65KB measured on device, giving the TLS handshake sufficient heap. The root cause was `MBEDTLS_ERR_X509_ALLOC_FAILED` (-0x2880) when a 3-cert chain consumed ~48KB during the handshake with only ~50KB available. - `KOReaderSyncActivity` no longer receives a `shared_ptr<Epub>` at construction — it lazy-loads the Epub after TLS only if remote progress is found (`ensureEpubLoaded()`). - Added `MIN_HEAP_FOR_TLS = 55000` guard in `KOReaderSyncClient` — returns `LOW_MEMORY` early if aggregate free heap is too low before attempting a TLS connection. **Navigation simplification (from #1760):** - Replaced `startActivityForResult` + callback with `activityManager.replaceActivity` / `activityManager.goToReader`. Progress is saved to `progress.bin` before the epub is released (cancel/upload paths) and in `saveProgressAndReturn` (apply remote path). The reader re-launches from the saved position naturally via `goToReader`, eliminating the need to reload the epub in a callback. - Extracted `ReaderUtils::saveProgress()` as a shared helper used by both `EpubReaderActivity` and `KOReaderSyncActivity`. - Added `STR_SAVE_PROGRESS_FAILED` to all 22 language files for the case where writing the synced position to SD fails. **Orientation fix (found during device testing):** - `EpubReaderActivity::onExit()` resets the renderer to portrait before destruction. With `replaceActivity` the reader is fully torn down before KOSync starts, so KOSync was always rendering in portrait even when reading in landscape. Fixed by calling `ReaderUtils::applyOrientation` in `KOReaderSyncActivity::onEnter()`. ## Additional Context Heap measurements on device (large EPUB with complex CSS): | Metric | Before | After | |---|---|---| | Heap before Epub release | 88,156 bytes | — | | Heap after Epub release | — | 153,892 bytes (+65,736) | | Heap at TLS handshake | ~50,000 bytes (fails) | ~116,384 bytes (passes) | | Min-free-ever during sync session | 2,600 bytes | 33,052 bytes | | TLS result | `MBEDTLS_ERR_X509_ALLOC_FAILED` | HTTP 200 | Tested on device: sync from inside a large EPUB in both portrait and landscape, cancel, apply remote progress, upload local progress. --- ### AI Usage Did you use AI tools to help write this code? _**YES**_ --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
|
14e1ce2d04 |
fix: use esp_http_client for KOSync to prevent TLS OOM on ESP32-C3 (#1381)
## Summary * **What is the goal of this PR?** Fix KOReader sync failing on HTTPS servers due to TLS out-of-memory on ESP32-C3 * **What changes are included?** - Replace `WiFiClientSecure`/`HTTPClient` with `esp_http_client` (ESP-IDF native API) for all KOSync HTTP requests - Use 2KB TLS buffers instead of the default 16KB — KOSync payloads are tiny JSON (<1KB), so this is more than sufficient - Use `esp_crt_bundle_attach` for proper TLS certificate verification (replaces `setInsecure()`) - Strip trailing slashes from server URL to prevent double-slash in API paths (e.g. `https://server.com//users/auth`) - Add `lastHttpCode` static field for diagnostics - Add free heap logging to help debug memory issues ### Problem The ESP32-C3 has ~46KB free heap after WiFi is initialized. `WiFiClientSecure` allocates 16KB for TLS RX + 16KB for TLS TX = 32KB just for the TLS buffers, leaving almost no room for the actual TLS handshake (which needs additional dynamic allocations). This causes KOReader sync to: 1. Fail silently with network errors on HTTPS servers (including the default `sync.koreader.rocks`) 2. Occasionally crash with heap exhaustion ### Solution `esp_http_client` (the ESP-IDF native HTTP client) allows configuring `buffer_size` and `buffer_size_tx` independently. Setting both to 2KB (total 4KB) leaves plenty of heap for the TLS handshake while still being sufficient for KOSync's small JSON payloads. This also fixes the `setInsecure()` anti-pattern — `esp_crt_bundle_attach` provides proper certificate verification using the ESP-IDF's built-in CA bundle, so credentials and reading history are no longer sent over unverified TLS connections. ### Files changed | File | Change | |------|--------| | `lib/KOReaderSync/KOReaderSyncClient.cpp` | Replace WiFiClientSecure/HTTPClient with esp_http_client; add ResponseBuffer, base64 encoder, createClient helper | | `lib/KOReaderSync/KOReaderSyncClient.h` | Add `lastHttpCode` static field | | `lib/KOReaderSync/KOReaderCredentialStore.cpp` | Strip trailing slashes from base URL | ## Additional Context Tested on a CrossPoint X4 (ESP32-C3 with 4MB flash). Before this change, KOSync auth to `sync.koreader.rocks` (HTTPS) would fail ~80% of the time. After: works reliably. The `base64Encode` helper is needed because `esp_http_client` doesn't have a built-in `setAuthorization()` method like `HTTPClient` does. This is used for the HTTP Basic Auth header required by Calibre-Web-Automated KOSync servers. Fixes #581 --- ### AI Usage While CrossPoint doesn't have restrictions on AI tools in contributing, please be transparent about their usage as it helps set the right context for reviewers. Did you use AI tools to help write this code? _**PARTIALLY**_ AI assisted with the esp_http_client migration pattern and base64 encoder. The root cause analysis (TLS buffer OOM) and solution design were manual. --------- Co-authored-by: trilwu <trilwu@users.noreply.github.com> Co-authored-by: Justin Mitchell <justin@jmitch.com> |
||
|
Justin Mitchellandjpirnay
|
302dea1eea |
fix: Switch to xpath map for paragraph level syncing in KOSync (#1686)
Switch KOReader sync progress mapping from chapter matching to XPath-based mapping. - resolves KOReader positions using real XHTML ancestry paths - supports paragraph-based upload mapping with text offsets where needed - passes the current paragraph index into sync so uploads map back to KOReader more accurately No HTTP client changes are included. No reader-state or resume-flow changes are included. --------- Co-authored-by: jpirnay <jens@pirnay.com> |
||
|
Zach Nelson
|
23aad213fc |
refactor: Removed redundant FsFile close() calls (#1434)
## Summary **What is the goal of this PR?** (e.g., Implements the new feature for file uploading.) `DESTRUCTOR_CLOSES_FILE=1` is set in platformio.ini, which makes SdFat's FsBaseFile destructor call close() automatically when a file goes out of scope. Three categories of file close calls remain untouched: 1. Close before Storage.remove() on the same path: ScreenshotUtil.cpp closes the file before deleting it on write error. The remove might fail if the file is still open. 2. Close before reopening the same variable: Epub.cpp writes a temp NCX/nav file, closes it, then reopens it for reading. The RecentBooksStore.cpp close before saveToFile() is the same pattern, it rewrites the same file. 3. Close on member variables: BookMetadataCache.cpp (bookFile, spineFile, tocFile), Section.cpp (file), XtcParser.cpp (m_file), ZipFile.cpp (file). These persist beyond any single function scope, so the destructor timing doesn't match the intended close point. --- ### AI Usage While CrossPoint doesn't have restrictions on AI tools in contributing, please be transparent about their usage as it helps set the right context for reviewers. Did you use AI tools to help write this code? _**PARTIALLY**_ |
||
|
jpirnayandDave Allie
|
6e4d0e534d |
feat: Migrate binary settings to json (#920)
## Summary
* This PR introduces a migration from binary file storage to JSON-based
storage for application settings, state, and various credential stores.
This improves readability, maintainability, and allows for easier manual
configuration editing.
* Benefits:
- Settings files are now JSON and can be easily read/edited manually
- Easier to inspect application state and settings during development
- JSON structure is more flexible for future changes
* Drawback: around 15k of additional flash usage
* Compatibility: Seamless migration preserves existing user data
## Additional Context
1. New JSON I/O Infrastructure files:
- JsonSettingsIO: Core JSON serialization/deserialization logic using
ArduinoJson library
- ObfuscationUtils: XOR-based password obfuscation for sensitive data
2. Migrated Components (now use JSON storage with automatic binary
migration):
- CrossPointSettings (settings.json): Main application settings
- CrossPointState (state.json): Application state (open book, sleep
mode, etc.)
- WifiCredentialStore (wifi.json): WiFi network credentials (Password
Obfuscation: Sensitive data like WiFi passwords, uses XOR encryption
with fixed keys. Note: This is obfuscation, not cryptographic security -
passwords can be recovered with the key)
- KOReaderCredentialStore (koreader.json): KOReader sync credentials
- RecentBooksStore (recent.json): Recently opened books list
3. Migration Logic
- Forward Compatibility: New installations use JSON format
- Backward Compatibility: Existing binary files are automatically
migrated to JSON on first load
- Backup Safety: Original binary files are renamed with .bak extension
after successful migration
- Fallback Handling: If JSON parsing fails, system falls back to binary
loading
4. Infrastructure Updates
- HalStorage: Added rename() method for backup operations
---
### AI Usage
While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.
Did you use AI tools to help write this code? _** YES**_
---------
Co-authored-by: Dave Allie <dave@daveallie.com>
|
||
|
jpirnay
|
00e25b1a90 |
fix: Fix kosync repositioning issue (#783)
## Summary * Original implementation had inconsistent positioning logic: - When XPath parsing succeeded: incorrectly set pageNumber = 0 (always beginning of chapter) - When XPath parsing failed: used percentage for positioning (worked correctly) - Result: Positions restored to wrong locations depending on XPath parsing success - Mentioned in Issue #581 * Solution - Unified ProgressMapper::toCrossPoint() to use percentage-based positioning exclusively for both spine identification and intra-chapter page calculation, eliminating unreliable XPath parsing entirely. ## Additional Context * ProgressMapper.cpp: Simplified toCrossPoint() to always use percentage for positioning, removed parseDocFragmentIndex() function * ProgressMapper.h: Updated comments and removed unused function declaration * Tests confirmed appropriate positioning * __Notabene: the syncing to another device will (most probably) end up at the current chapter of crosspoints reading position. There is not much we can do about it, as KOReader needs to have the correct XPath information - we can only provide an apporximate position (plus percentage) - the percentage information is not used in KOReaders current implementation__ --- ### AI Usage While CrossPoint doesn't have restrictions on AI tools in contributing, please be transparent about their usage as it helps set the right context for reviewers. Did you use AI tools to help write this code? YES |
||
|
jpirnayandXuan Son Nguyen
|
cb24947477 |
feat: Add central logging pragma (#843)
## Summary
* Definition and use of a central LOG function, that can later be
extended or completely be removed (for public use where debugging
information may not be required) to save flash by suppressing the
-DENABLE_SERIAL_LOG like in the slim branch
* **What changes are included?**
## Additional Context
* By using the central logger the usual:
```
#include <HardwareSerial.h>
...
Serial.printf("[%lu] [WCS] Obfuscating/deobfuscating %zu bytes\n", millis(), data.size());
```
would then become
```
#include <Logging.h>
...
LOG_DBG("WCS", "Obfuscating/deobfuscating %zu bytes", data.size());
```
You do have ``LOG_DBG`` for debug messages, ``LOG_ERR`` for error
messages and ``LOG_INF`` for informational messages. Depending on the
verbosity level defined (see below) soe of these message types will be
suppressed/not-compiled.
* The normal compilation (default) will create a firmware.elf file of
42.194.356 bytes, the same code via slim will create 42.024.048 bytes -
170.308 bytes less
* Firmware.bin : 6.469.984 bytes for default, 6.418.672 bytes for slim -
51.312 bytes less
### AI Usage
While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.
Did you use AI tools to help write this code? _NO_
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
|
||
|
Xuan-Son Nguyen
|
7f40c3f477 |
feat: add HalStorage (#656)
## Summary Continue my changes to introduce the HAL infrastructure from https://github.com/crosspoint-reader/crosspoint-reader/pull/522 This PR touches quite a lot of files, but most of them are just name changing. It should not have any impacts to the end behavior. ## Additional Context My plan is to firstly add this small shim layer, which sounds useless at first, but then I'll implement an emulated driver which can be helpful for testing and for development. Currently, on my fork, I'm using a FS driver that allow "mounting" a local directory from my computer to the device, much like the `-v` mount option on docker. This allows me to quickly reset `.crosspoint` directory if anything goes wrong. I plan to upstream this feature when this PR get merged. --- ### AI Usage While CrossPoint doesn't have restrictions on AI tools in contributing, please be transparent about their usage as it helps set the right context for reviewers. Did you use AI tools to help write this code? NO |
||
|
Fabio Barbonanddrbourbon
|
17fedd2a69 |
feat: Calibre Web Automated (CWA) koreader sync server support (#594)
## Summary * **What is the goal of this PR?** Provide support to koreader sync server embedded in the popular *Calibre Web Automated* self-hosted digital library solution. * **What changes are included?** * Trivial addition of **HTTP Basic Auth (RFC 7617) header** to `lib/KOReaderSync/KOReaderSyncClient.cpp` ## Additional Context None --- ### AI Usage While CrossPoint doesn't have restrictions on AI tools in contributing, please be transparent about their usage as it helps set the right context for reviewers. Did you use AI tools to help write this code? _**NO**_ --------- Co-authored-by: drbourbon <fabio@MacBook-Air-di-Fabio.local> |
||
|
Carson Hicks
|
dfd7b615dc |
fix: Fix KOReader document md5 calculation for binary matching progress sync (#529)
## Summary * **What is the goal of this PR?** Resolve [KoSync progress does not sync between Crosspoint-reader and KOReader (Kindle)](https://github.com/crosspoint-reader/crosspoint-reader/issues/502) * **What changes are included?** KOReaderDocumentId::getOffset() - Update the value for the md5 offset calculation to match KOReader. ## Additional Context I've tested this with a couple of my ebooks and binary matching with KOReader sync seems to be working fine now for both pushing and pulling progress. --- ### AI Usage While CrossPoint doesn't have restrictions on AI tools in contributing, please be transparent about their usage as it helps set the right context for reviewers. Did you use AI tools to help write this code? _**NO**_ |
||
|
Justin MitchellandDave Allie
|
f69cddf2cc |
Adds KOReader Sync support (#232)
## Summary - Adds KOReader progress sync integration, allowing CrossPoint to sync reading positions with other KOReader-compatible devices - Stores credentials securely with XOR obfuscation - Uses KOReader's partial MD5 document hashing for cross-device book matching - Syncs position via percentage with estimated XPath for compatibility # Features - Settings: KOReader Username, Password, and Authenticate options - Sync from chapters menu: "Sync Progress" option appears when credentials are configured - Bidirectional sync: Can apply remote progress or upload local progress --------- Co-authored-by: Dave Allie <dave@daveallie.com> |