The Arduino WiFiClientSecure allocates 16KB TLS buffers by default,
which exhausts the ESP32-C3's limited heap (~46KB free after WiFi)
during the TLS handshake. This causes KOReader sync to fail silently
or crash on HTTPS servers (including the default sync.koreader.rocks).
Replace WiFiClientSecure/HTTPClient with esp_http_client (ESP-IDF),
which supports configurable buffer sizes. Use 2KB TLS buffers — more
than sufficient for KOSync's tiny JSON payloads (<1KB).
Also:
- Use esp_crt_bundle for proper TLS certificate verification instead
of setInsecure()
- Strip trailing slashes from server URL to prevent double-slash in
API paths
- Add lastHttpCode for diagnostics
- Add heap logging to help debug memory issues
Fixes#581
(cherry picked from commit 835abc19ff14fcbea8571ccb0dfd8d21e882b84c)
Store per-page paragraph indices in section cache to enable precise
XPath-to-page and page-to-XPath mapping without reparsing XHTML.
Forward path (upload): generates XPath directly from paragraph LUT
instead of byte-offset estimation, eliminating drift in chapters
with non-uniform content density.
Reverse path (download): resolves incoming KOReader XPath p[N] to
the exact page via paragraph LUT lookup.
Paragraph counter counts all <p> elements including display:none
to match ChapterXPathIndexer and crengine's standard XPath counting.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Store per-page paragraph indices in section cache to enable precise
XPath-to-page and page-to-XPath mapping without reparsing XHTML.
Forward path (upload): generates XPath directly from paragraph LUT
instead of byte-offset estimation, eliminating drift in chapters
with non-uniform content density.
Reverse path (download): resolves incoming KOReader XPath p[N] to
the exact page via paragraph LUT lookup.
Paragraph counter counts all <p> elements including display:none
to match ChapterXPathIndexer and crengine's standard XPath counting.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
* This PR introduces a migration from binary file storage to JSON-based
storage for application settings, state, and various credential stores.
This improves readability, maintainability, and allows for easier manual
configuration editing.
* Benefits:
- Settings files are now JSON and can be easily read/edited manually
- Easier to inspect application state and settings during development
- JSON structure is more flexible for future changes
* Drawback: around 15k of additional flash usage
* Compatibility: Seamless migration preserves existing user data
## Additional Context
1. New JSON I/O Infrastructure files:
- JsonSettingsIO: Core JSON serialization/deserialization logic using
ArduinoJson library
- ObfuscationUtils: XOR-based password obfuscation for sensitive data
2. Migrated Components (now use JSON storage with automatic binary
migration):
- CrossPointSettings (settings.json): Main application settings
- CrossPointState (state.json): Application state (open book, sleep
mode, etc.)
- WifiCredentialStore (wifi.json): WiFi network credentials (Password
Obfuscation: Sensitive data like WiFi passwords, uses XOR encryption
with fixed keys. Note: This is obfuscation, not cryptographic security -
passwords can be recovered with the key)
- KOReaderCredentialStore (koreader.json): KOReader sync credentials
- RecentBooksStore (recent.json): Recently opened books list
3. Migration Logic
- Forward Compatibility: New installations use JSON format
- Backward Compatibility: Existing binary files are automatically
migrated to JSON on first load
- Backup Safety: Original binary files are renamed with .bak extension
after successful migration
- Fallback Handling: If JSON parsing fails, system falls back to binary
loading
4. Infrastructure Updates
- HalStorage: Added rename() method for backup operations
---
### AI Usage
While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.
Did you use AI tools to help write this code? _** YES**_
---------
Co-authored-by: Dave Allie <dave@daveallie.com>
## Summary
* Original implementation had inconsistent positioning logic:
- When XPath parsing succeeded: incorrectly set pageNumber = 0 (always
beginning of chapter)
- When XPath parsing failed: used percentage for positioning (worked
correctly)
- Result: Positions restored to wrong locations depending on XPath
parsing success
- Mentioned in Issue #581
* Solution
- Unified ProgressMapper::toCrossPoint() to use percentage-based
positioning exclusively for both spine identification and intra-chapter
page calculation, eliminating unreliable XPath parsing entirely.
## Additional Context
* ProgressMapper.cpp: Simplified toCrossPoint() to always use percentage
for positioning, removed parseDocFragmentIndex() function
* ProgressMapper.h: Updated comments and removed unused function
declaration
* Tests confirmed appropriate positioning
* __Notabene: the syncing to another device will (most probably) end up
at the current chapter of crosspoints reading position. There is not
much we can do about it, as KOReader needs to have the correct XPath
information - we can only provide an apporximate position (plus
percentage) - the percentage information is not used in KOReaders
current implementation__
---
### AI Usage
While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.
Did you use AI tools to help write this code? YES
## Summary
* Definition and use of a central LOG function, that can later be
extended or completely be removed (for public use where debugging
information may not be required) to save flash by suppressing the
-DENABLE_SERIAL_LOG like in the slim branch
* **What changes are included?**
## Additional Context
* By using the central logger the usual:
```
#include <HardwareSerial.h>
...
Serial.printf("[%lu] [WCS] Obfuscating/deobfuscating %zu bytes\n", millis(), data.size());
```
would then become
```
#include <Logging.h>
...
LOG_DBG("WCS", "Obfuscating/deobfuscating %zu bytes", data.size());
```
You do have ``LOG_DBG`` for debug messages, ``LOG_ERR`` for error
messages and ``LOG_INF`` for informational messages. Depending on the
verbosity level defined (see below) soe of these message types will be
suppressed/not-compiled.
* The normal compilation (default) will create a firmware.elf file of
42.194.356 bytes, the same code via slim will create 42.024.048 bytes -
170.308 bytes less
* Firmware.bin : 6.469.984 bytes for default, 6.418.672 bytes for slim -
51.312 bytes less
### AI Usage
While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.
Did you use AI tools to help write this code? _NO_
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
jpirnay