## Summary
Make KOReader progress sync a one-click flow for common cases while
keeping the existing manual mode available.
### Changes
- Add a **Sync Behavior** setting:
- **Smart sync** for new configurations
- **Ask every time** for manual control
- Preserve **Ask every time** when migrating an existing credential file
that predates this setting.
- In Smart mode:
- upload local progress when no remote record exists;
- show a short confirmation when already synced;
- upload when local progress is further ahead;
- apply remote progress when remote progress is further ahead.
- Probe both KOReader document-matching hashes before making the Smart
decision, while keeping uploads on the user's configured matching
method.
- Auto-return after successful Smart terminal states, with Back/Confirm
still available.
- Persist the setting alongside the current credential, matching-method,
and send-metadata fields.
- Document both behaviors in the user guide.
## Additional context
This pairs well with #2189 (smart Wi-Fi auto-connect), but does not
depend on it. It does not add background Wi-Fi or passive network
detection; sync is still triggered by the user from the reader menu.
Smart sync uses the furthest progress because CrossPoint does not
currently persist local progress timestamps. Users who prefer explicit
conflict resolution can select **Ask every time**.
## Verification
- `git diff --check`
- `platformio check --fail-on-defect low --fail-on-defect medium
--fail-on-defect high` — no defects
- `platformio run -e default` — firmware build successful
- The original implementation was manually smoke-tested on an X4 device.
---
### AI Usage
Did you use AI tools to help write this code? _**YES**_
AI tools were used to inspect the codebase, draft and review the
implementation, resolve the rebase against current `develop`, and
prepare the PR text. The resulting firmware was built locally.
---------
Co-authored-by: Alexander Hoffer <git@alexanderhoffer.com>
Co-authored-by: Alexander Hoffer <contact@alexanderhoffer.com>
## Summary
* **What is the goal of this PR?** Fix regression of
Calibre-Web-Automated authentication
* **What changes are included?** Trivial fix using ESP HTTP Client
primitives
## Additional Context
Calibre-Web-Automated KOSync stopped working after refactoring HTTP
client to use `ESP HTTP Client`, resulting in a _Server error (try again
later)_ error.
---
### AI Usage
Did you use AI tools to help write this code? _**NO**_
---------
Co-authored-by: drbourbon <fabio@MacBook-Air-di-Fabio.local>
## Summary
Relative includes can hide inappropriate dependency relationships. In
this case, I found that lib/KOReaderSync/KOReaderCredentialStore.cpp was
dependent on src/JsonSettingsIO.h -- a lib -> app dependency, the
opposite direction dependencies should flow in this project.
This change replaces all relative includes with root-relative includes,
and corrects the KOReaderCredentialStore dependency by moving its JSON
settings serialization local to the KOReaderSync library.
---
### AI Usage
While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.
Did you use AI tools to help write this code? _**PARTIALLY**_
## Summary
**Goal:** Fix bidirectional KOSync position matching between CrossPoint
and KOReader so that syncing in either direction lands on the correct
page with character-level accuracy.
**Changes included:**
**Download — `toCrossPoint` (server XPath → CrossPoint page)**
- **XPath ancestry mode for structured elements**: The previous
`ParagraphStreamer` only tracked `<p>` elements. Replaced with a full
ancestor-walking mode that correctly resolves XPaths pointing into
`<li>`, `<ul>`, and other structured elements. Char offset within the
target element is bounded to the matched element's content only.
- **Slash-in-attribute-value corrupts depth tracking**:
`processByteInTag()` treated every `/` byte as a self-closing tag
marker, including `/` inside quoted attribute values (e.g.
`xmlns="http://..."`, `src="Links/image.jpg"`). This drove `htmlDepth`
to 0 prematurely, causing the ancestry search to exit far short of the
target paragraph. Fixed with `inAttrQuote` tracking.
- **Off-by-one in page formula**: `intra * totalPages` rounds up
incorrectly for last-page positions. Changed to `intra * (totalPages -
1)` to map the `[0, 1]` intra fraction correctly onto the `[0,
totalPages-1]` page range. Example: page 14 of 17 was returned as 15.
**Upload — `toKOReader` (CrossPoint page → server XPath)**
- **Off-by-one in page-to-intra formula**: Symmetric fix — `pageNumber /
totalPages` changed to `pageNumber / (totalPages - 1)`, with the guard
updated from `> 0` to `> 1` to avoid division by zero.
- **`<li>`-based XPath generation**: When the current page starts on a
list item, `findXPathForProgress` now generates `ul[N]/li[M]` XPaths
rather than falling back to the preceding `<p>`. Requires the new
`listItemIndex` field in `PageLutEntry` (section cache version bumped to
23).
- **Text-node precision with correct `text()[N].M` format**: KOReader
expects `text()[N].M` where `N` is the 1-based index of the specific
text node within the element. The previous attempt generated `text().M`
(no brackets), which caused KOReader to jump to the front of the book.
Implements a per-element text-node index stack in
`XPathProgressResolver` — parallel to the existing element path stack —
that correctly tracks text node indices relative to each element. Empty
text nodes from bare anchor elements (`<a id="anchor"/>`) are
intentionally skipped, matching KOReader's own text node counting
behavior.
**Reviewer-caught bugs**
- **Double `onCloseTag()` on malformed `</br/>`**: Both the `tagIsClose`
path and the self-closing `/` check were firing, double-decrementing
`htmlDepth`. Fixed with a `!tagIsClose` guard.
- **Dangling pointer in `LOG_DBG`**:
`std::to_string(*nextParagraphPage).c_str()` passed a pointer to a
temporary destroyed before the variadic call. Fixed with `snprintf` into
a stack `char[8]` buffer.
## Additional Context
- Section cache version bumped from 22 → 23 due to the new
`listItemIndex` field in `PageLutEntry`. Users upgrading will see a
one-time re-render of all cached sections on first load — no data loss.
- The `textNodeIndexStack` in `XPathProgressResolver` is a
`std::vector<int>` that mirrors the existing `path` and `parentStates`
stacks — same depth, same lifetime. No additional heap pressure beyond
what was already present.
- All fixes verified on device with *Gentle and Lowly* by Dane C.
Ortlund (spine 21, 17 pages). Download syncs land on the correct page;
upload syncs land at the correct paragraph with character-level offset.
## Test plan
- [ ] Download: sync from KOReader → CrossPoint lands on correct page
for `text()[N].M` XPaths
- [ ] Download: ancestry correctly resolves `<li>` positions inbound
from KOReader
- [ ] Upload: sync from CrossPoint → KOReader lands within one page for
mid-paragraph positions
- [ ] Upload: sync from CrossPoint → KOReader correctly targets `<li>`
elements when page starts on a list item
- [ ] Upload: `text()[N].M` format XPaths do not cause KOReader to jump
to front of book
- [ ] Section cache version 23: delete `.crosspoint/` and verify clean
re-parse with no crashes
---
### AI Usage
Did you use AI tools to help write this code? **YES** — developed with
Claude Code (Anthropic).
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
* **What is the goal of this PR?** Fix KOSync failing with "Network
error" on large/complex EPUBs, and simplify the sync navigation flow by
removing the callback/result pattern.
* **What changes are included?**
This PR combines the approaches from #1855 and #1760 into a single,
cleaner solution:
**Memory fix (from #1855):**
- `EpubReaderActivity` pre-computes the local KOReader position and
chapter name, then explicitly releases `epub` and `section` before
launching `KOReaderSyncActivity`. This frees ~65KB measured on device,
giving the TLS handshake sufficient heap. The root cause was
`MBEDTLS_ERR_X509_ALLOC_FAILED` (-0x2880) when a 3-cert chain consumed
~48KB during the handshake with only ~50KB available.
- `KOReaderSyncActivity` no longer receives a `shared_ptr<Epub>` at
construction — it lazy-loads the Epub after TLS only if remote progress
is found (`ensureEpubLoaded()`).
- Added `MIN_HEAP_FOR_TLS = 55000` guard in `KOReaderSyncClient` —
returns `LOW_MEMORY` early if aggregate free heap is too low before
attempting a TLS connection.
**Navigation simplification (from #1760):**
- Replaced `startActivityForResult` + callback with
`activityManager.replaceActivity` / `activityManager.goToReader`.
Progress is saved to `progress.bin` before the epub is released
(cancel/upload paths) and in `saveProgressAndReturn` (apply remote
path). The reader re-launches from the saved position naturally via
`goToReader`, eliminating the need to reload the epub in a callback.
- Extracted `ReaderUtils::saveProgress()` as a shared helper used by
both `EpubReaderActivity` and `KOReaderSyncActivity`.
- Added `STR_SAVE_PROGRESS_FAILED` to all 22 language files for the case
where writing the synced position to SD fails.
**Orientation fix (found during device testing):**
- `EpubReaderActivity::onExit()` resets the renderer to portrait before
destruction. With `replaceActivity` the reader is fully torn down before
KOSync starts, so KOSync was always rendering in portrait even when
reading in landscape. Fixed by calling `ReaderUtils::applyOrientation`
in `KOReaderSyncActivity::onEnter()`.
## Additional Context
Heap measurements on device (large EPUB with complex CSS):
| Metric | Before | After |
|---|---|---|
| Heap before Epub release | 88,156 bytes | — |
| Heap after Epub release | — | 153,892 bytes (+65,736) |
| Heap at TLS handshake | ~50,000 bytes (fails) | ~116,384 bytes
(passes) |
| Min-free-ever during sync session | 2,600 bytes | 33,052 bytes |
| TLS result | `MBEDTLS_ERR_X509_ALLOC_FAILED` | HTTP 200 |
Tested on device: sync from inside a large EPUB in both portrait and
landscape, cancel, apply remote progress, upload local progress.
---
### AI Usage
Did you use AI tools to help write this code? _**YES**_
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
* **What is the goal of this PR?** Fix KOReader sync failing on HTTPS
servers due to TLS out-of-memory on ESP32-C3
* **What changes are included?**
- Replace `WiFiClientSecure`/`HTTPClient` with `esp_http_client`
(ESP-IDF native API) for all KOSync HTTP requests
- Use 2KB TLS buffers instead of the default 16KB — KOSync payloads are
tiny JSON (<1KB), so this is more than sufficient
- Use `esp_crt_bundle_attach` for proper TLS certificate verification
(replaces `setInsecure()`)
- Strip trailing slashes from server URL to prevent double-slash in API
paths (e.g. `https://server.com//users/auth`)
- Add `lastHttpCode` static field for diagnostics
- Add free heap logging to help debug memory issues
### Problem
The ESP32-C3 has ~46KB free heap after WiFi is initialized.
`WiFiClientSecure` allocates 16KB for TLS RX + 16KB for TLS TX = 32KB
just for the TLS buffers, leaving almost no room for the actual TLS
handshake (which needs additional dynamic allocations). This causes
KOReader sync to:
1. Fail silently with network errors on HTTPS servers (including the
default `sync.koreader.rocks`)
2. Occasionally crash with heap exhaustion
### Solution
`esp_http_client` (the ESP-IDF native HTTP client) allows configuring
`buffer_size` and `buffer_size_tx` independently. Setting both to 2KB
(total 4KB) leaves plenty of heap for the TLS handshake while still
being sufficient for KOSync's small JSON payloads.
This also fixes the `setInsecure()` anti-pattern —
`esp_crt_bundle_attach` provides proper certificate verification using
the ESP-IDF's built-in CA bundle, so credentials and reading history are
no longer sent over unverified TLS connections.
### Files changed
| File | Change |
|------|--------|
| `lib/KOReaderSync/KOReaderSyncClient.cpp` | Replace
WiFiClientSecure/HTTPClient with esp_http_client; add ResponseBuffer,
base64 encoder, createClient helper |
| `lib/KOReaderSync/KOReaderSyncClient.h` | Add `lastHttpCode` static
field |
| `lib/KOReaderSync/KOReaderCredentialStore.cpp` | Strip trailing
slashes from base URL |
## Additional Context
Tested on a CrossPoint X4 (ESP32-C3 with 4MB flash). Before this change,
KOSync auth to `sync.koreader.rocks` (HTTPS) would fail ~80% of the
time. After: works reliably.
The `base64Encode` helper is needed because `esp_http_client` doesn't
have a built-in `setAuthorization()` method like `HTTPClient` does. This
is used for the HTTP Basic Auth header required by Calibre-Web-Automated
KOSync servers.
Fixes#581
---
### AI Usage
While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it helps set the right
context for reviewers.
Did you use AI tools to help write this code? _**PARTIALLY**_ AI
assisted with the esp_http_client migration pattern and base64 encoder.
The root cause analysis (TLS buffer OOM) and solution design were
manual.
---------
Co-authored-by: trilwu <trilwu@users.noreply.github.com>
Co-authored-by: Justin Mitchell <justin@jmitch.com>
Switch KOReader sync progress mapping from chapter matching to
XPath-based mapping.
- resolves KOReader positions using real XHTML ancestry paths
- supports paragraph-based upload mapping with text offsets
where needed
- passes the current paragraph index into sync so uploads map
back to KOReader more accurately
No HTTP client changes are included. No reader-state or resume-flow
changes are included.
---------
Co-authored-by: jpirnay <jens@pirnay.com>
## Summary
**What is the goal of this PR?** (e.g., Implements the new feature for
file uploading.)
`DESTRUCTOR_CLOSES_FILE=1` is set in platformio.ini, which makes SdFat's
FsBaseFile destructor call close() automatically when a file goes out of
scope.
Three categories of file close calls remain untouched:
1. Close before Storage.remove() on the same path: ScreenshotUtil.cpp
closes the file before deleting it on write error. The remove might fail
if the file is still open.
2. Close before reopening the same variable: Epub.cpp writes a temp
NCX/nav file, closes it, then reopens it for reading. The
RecentBooksStore.cpp close before saveToFile() is the same pattern, it
rewrites the same file.
3. Close on member variables: BookMetadataCache.cpp (bookFile,
spineFile, tocFile), Section.cpp (file), XtcParser.cpp (m_file),
ZipFile.cpp (file). These persist beyond any single function scope, so
the destructor timing doesn't match the intended close point.
---
### AI Usage
While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.
Did you use AI tools to help write this code? _**PARTIALLY**_
## Summary
* This PR introduces a migration from binary file storage to JSON-based
storage for application settings, state, and various credential stores.
This improves readability, maintainability, and allows for easier manual
configuration editing.
* Benefits:
- Settings files are now JSON and can be easily read/edited manually
- Easier to inspect application state and settings during development
- JSON structure is more flexible for future changes
* Drawback: around 15k of additional flash usage
* Compatibility: Seamless migration preserves existing user data
## Additional Context
1. New JSON I/O Infrastructure files:
- JsonSettingsIO: Core JSON serialization/deserialization logic using
ArduinoJson library
- ObfuscationUtils: XOR-based password obfuscation for sensitive data
2. Migrated Components (now use JSON storage with automatic binary
migration):
- CrossPointSettings (settings.json): Main application settings
- CrossPointState (state.json): Application state (open book, sleep
mode, etc.)
- WifiCredentialStore (wifi.json): WiFi network credentials (Password
Obfuscation: Sensitive data like WiFi passwords, uses XOR encryption
with fixed keys. Note: This is obfuscation, not cryptographic security -
passwords can be recovered with the key)
- KOReaderCredentialStore (koreader.json): KOReader sync credentials
- RecentBooksStore (recent.json): Recently opened books list
3. Migration Logic
- Forward Compatibility: New installations use JSON format
- Backward Compatibility: Existing binary files are automatically
migrated to JSON on first load
- Backup Safety: Original binary files are renamed with .bak extension
after successful migration
- Fallback Handling: If JSON parsing fails, system falls back to binary
loading
4. Infrastructure Updates
- HalStorage: Added rename() method for backup operations
---
### AI Usage
While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.
Did you use AI tools to help write this code? _** YES**_
---------
Co-authored-by: Dave Allie <dave@daveallie.com>
## Summary
* Original implementation had inconsistent positioning logic:
- When XPath parsing succeeded: incorrectly set pageNumber = 0 (always
beginning of chapter)
- When XPath parsing failed: used percentage for positioning (worked
correctly)
- Result: Positions restored to wrong locations depending on XPath
parsing success
- Mentioned in Issue #581
* Solution
- Unified ProgressMapper::toCrossPoint() to use percentage-based
positioning exclusively for both spine identification and intra-chapter
page calculation, eliminating unreliable XPath parsing entirely.
## Additional Context
* ProgressMapper.cpp: Simplified toCrossPoint() to always use percentage
for positioning, removed parseDocFragmentIndex() function
* ProgressMapper.h: Updated comments and removed unused function
declaration
* Tests confirmed appropriate positioning
* __Notabene: the syncing to another device will (most probably) end up
at the current chapter of crosspoints reading position. There is not
much we can do about it, as KOReader needs to have the correct XPath
information - we can only provide an apporximate position (plus
percentage) - the percentage information is not used in KOReaders
current implementation__
---
### AI Usage
While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.
Did you use AI tools to help write this code? YES
## Summary
* Definition and use of a central LOG function, that can later be
extended or completely be removed (for public use where debugging
information may not be required) to save flash by suppressing the
-DENABLE_SERIAL_LOG like in the slim branch
* **What changes are included?**
## Additional Context
* By using the central logger the usual:
```
#include <HardwareSerial.h>
...
Serial.printf("[%lu] [WCS] Obfuscating/deobfuscating %zu bytes\n", millis(), data.size());
```
would then become
```
#include <Logging.h>
...
LOG_DBG("WCS", "Obfuscating/deobfuscating %zu bytes", data.size());
```
You do have ``LOG_DBG`` for debug messages, ``LOG_ERR`` for error
messages and ``LOG_INF`` for informational messages. Depending on the
verbosity level defined (see below) soe of these message types will be
suppressed/not-compiled.
* The normal compilation (default) will create a firmware.elf file of
42.194.356 bytes, the same code via slim will create 42.024.048 bytes -
170.308 bytes less
* Firmware.bin : 6.469.984 bytes for default, 6.418.672 bytes for slim -
51.312 bytes less
### AI Usage
While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.
Did you use AI tools to help write this code? _NO_
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
## Summary
Continue my changes to introduce the HAL infrastructure from
https://github.com/crosspoint-reader/crosspoint-reader/pull/522
This PR touches quite a lot of files, but most of them are just name
changing. It should not have any impacts to the end behavior.
## Additional Context
My plan is to firstly add this small shim layer, which sounds useless at
first, but then I'll implement an emulated driver which can be helpful
for testing and for development.
Currently, on my fork, I'm using a FS driver that allow "mounting" a
local directory from my computer to the device, much like the `-v` mount
option on docker. This allows me to quickly reset `.crosspoint`
directory if anything goes wrong. I plan to upstream this feature when
this PR get merged.
---
### AI Usage
While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.
Did you use AI tools to help write this code? NO
## Summary
* **What is the goal of this PR?** Provide support to koreader sync
server embedded in the popular *Calibre Web Automated* self-hosted
digital library solution.
* **What changes are included?**
* Trivial addition of **HTTP Basic Auth (RFC 7617) header** to
`lib/KOReaderSync/KOReaderSyncClient.cpp`
## Additional Context
None
---
### AI Usage
While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.
Did you use AI tools to help write this code? _**NO**_
---------
Co-authored-by: drbourbon <fabio@MacBook-Air-di-Fabio.local>
## Summary
* **What is the goal of this PR?**
Resolve [KoSync progress does not sync between Crosspoint-reader and
KOReader
(Kindle)](https://github.com/crosspoint-reader/crosspoint-reader/issues/502)
* **What changes are included?**
KOReaderDocumentId::getOffset() - Update the value for the md5 offset
calculation to match KOReader.
## Additional Context
I've tested this with a couple of my ebooks and binary matching with
KOReader sync seems to be working fine now for both pushing and pulling
progress.
---
### AI Usage
While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.
Did you use AI tools to help write this code? _**NO**_
## Summary
- Adds KOReader progress sync integration, allowing CrossPoint to sync
reading positions with other
KOReader-compatible devices
- Stores credentials securely with XOR obfuscation
- Uses KOReader's partial MD5 document hashing for cross-device book
matching
- Syncs position via percentage with estimated XPath for compatibility
# Features
- Settings: KOReader Username, Password, and Authenticate options
- Sync from chapters menu: "Sync Progress" option appears when
credentials are configured
- Bidirectional sync: Can apply remote progress or upload local progress
---------
Co-authored-by: Dave Allie <dave@daveallie.com>
