fix: improve KOSync bidirectional position matching accuracy (#1897)

## Summary

**Goal:** Fix bidirectional KOSync position matching between CrossPoint
and KOReader so that syncing in either direction lands on the correct
page with character-level accuracy.

**Changes included:**

**Download — `toCrossPoint` (server XPath → CrossPoint page)**
- **XPath ancestry mode for structured elements**: The previous
`ParagraphStreamer` only tracked `<p>` elements. Replaced with a full
ancestor-walking mode that correctly resolves XPaths pointing into
`<li>`, `<ul>`, and other structured elements. Char offset within the
target element is bounded to the matched element's content only.
- **Slash-in-attribute-value corrupts depth tracking**:
`processByteInTag()` treated every `/` byte as a self-closing tag
marker, including `/` inside quoted attribute values (e.g.
`xmlns="http://..."`, `src="Links/image.jpg"`). This drove `htmlDepth`
to 0 prematurely, causing the ancestry search to exit far short of the
target paragraph. Fixed with `inAttrQuote` tracking.
- **Off-by-one in page formula**: `intra * totalPages` rounds up
incorrectly for last-page positions. Changed to `intra * (totalPages -
1)` to map the `[0, 1]` intra fraction correctly onto the `[0,
totalPages-1]` page range. Example: page 14 of 17 was returned as 15.

**Upload — `toKOReader` (CrossPoint page → server XPath)**
- **Off-by-one in page-to-intra formula**: Symmetric fix — `pageNumber /
totalPages` changed to `pageNumber / (totalPages - 1)`, with the guard
updated from `> 0` to `> 1` to avoid division by zero.
- **`<li>`-based XPath generation**: When the current page starts on a
list item, `findXPathForProgress` now generates `ul[N]/li[M]` XPaths
rather than falling back to the preceding `<p>`. Requires the new
`listItemIndex` field in `PageLutEntry` (section cache version bumped to
23).
- **Text-node precision with correct `text()[N].M` format**: KOReader
expects `text()[N].M` where `N` is the 1-based index of the specific
text node within the element. The previous attempt generated `text().M`
(no brackets), which caused KOReader to jump to the front of the book.
Implements a per-element text-node index stack in
`XPathProgressResolver` — parallel to the existing element path stack —
that correctly tracks text node indices relative to each element. Empty
text nodes from bare anchor elements (`<a id="anchor"/>`) are
intentionally skipped, matching KOReader's own text node counting
behavior.

**Reviewer-caught bugs**
- **Double `onCloseTag()` on malformed `</br/>`**: Both the `tagIsClose`
path and the self-closing `/` check were firing, double-decrementing
`htmlDepth`. Fixed with a `!tagIsClose` guard.
- **Dangling pointer in `LOG_DBG`**:
`std::to_string(*nextParagraphPage).c_str()` passed a pointer to a
temporary destroyed before the variadic call. Fixed with `snprintf` into
a stack `char[8]` buffer.

## Additional Context

- Section cache version bumped from 22 → 23 due to the new
`listItemIndex` field in `PageLutEntry`. Users upgrading will see a
one-time re-render of all cached sections on first load — no data loss.
- The `textNodeIndexStack` in `XPathProgressResolver` is a
`std::vector<int>` that mirrors the existing `path` and `parentStates`
stacks — same depth, same lifetime. No additional heap pressure beyond
what was already present.
- All fixes verified on device with *Gentle and Lowly* by Dane C.
Ortlund (spine 21, 17 pages). Download syncs land on the correct page;
upload syncs land at the correct paragraph with character-level offset.

## Test plan

- [ ] Download: sync from KOReader → CrossPoint lands on correct page
for `text()[N].M` XPaths
- [ ] Download: ancestry correctly resolves `<li>` positions inbound
from KOReader
- [ ] Upload: sync from CrossPoint → KOReader lands within one page for
mid-paragraph positions
- [ ] Upload: sync from CrossPoint → KOReader correctly targets `<li>`
elements when page starts on a list item
- [ ] Upload: `text()[N].M` format XPaths do not cause KOReader to jump
to front of book
- [ ] Section cache version 23: delete `.crosspoint/` and verify clean
re-parse with no crashes

---

### AI Usage

Did you use AI tools to help write this code? **YES** — developed with
Claude Code (Anthropic).

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Wylan Swets
2026-05-09 12:22:24 -04:00
committed by GitHub
co-authored by Claude Sonnet 4.6
parent 628794f8b8
commit bf894fd343
8 changed files with 585 additions and 103 deletions
+68 -11
View File
@@ -10,14 +10,16 @@
#include "parsers/ChapterHtmlSlimParser.h"
namespace {
constexpr uint8_t SECTION_FILE_VERSION = 22;
constexpr uint8_t SECTION_FILE_VERSION = 23;
constexpr uint32_t HEADER_SIZE = sizeof(uint8_t) + sizeof(int) + sizeof(float) + sizeof(bool) + sizeof(uint8_t) +
sizeof(uint16_t) + sizeof(uint16_t) + sizeof(uint16_t) + sizeof(bool) + sizeof(bool) +
sizeof(uint8_t) + sizeof(uint32_t) + sizeof(uint32_t) + sizeof(uint32_t);
sizeof(uint8_t) + sizeof(uint32_t) + sizeof(uint32_t) + sizeof(uint32_t) +
sizeof(uint32_t);
struct PageLutEntry {
uint32_t fileOffset;
uint16_t paragraphIndex;
uint16_t listItemIndex;
};
} // namespace
@@ -50,7 +52,7 @@ void Section::writeSectionFileHeader(const int fontId, const float lineCompressi
sizeof(extraParagraphSpacing) + sizeof(paragraphAlignment) + sizeof(viewportWidth) +
sizeof(viewportHeight) + sizeof(pageCount) + sizeof(hyphenationEnabled) +
sizeof(embeddedStyle) + sizeof(imageRendering) + sizeof(uint32_t) +
sizeof(uint32_t) + sizeof(uint32_t),
sizeof(uint32_t) + sizeof(uint32_t) + sizeof(uint32_t),
"Header size mismatch");
serialization::writePod(file, SECTION_FILE_VERSION);
serialization::writePod(file, fontId);
@@ -66,6 +68,7 @@ void Section::writeSectionFileHeader(const int fontId, const float lineCompressi
serialization::writePod(file, static_cast<uint32_t>(0)); // Placeholder for LUT offset (patched later)
serialization::writePod(file, static_cast<uint32_t>(0)); // Placeholder for anchor map offset (patched later)
serialization::writePod(file, static_cast<uint32_t>(0)); // Placeholder for paragraph LUT offset (patched later)
serialization::writePod(file, static_cast<uint32_t>(0)); // Placeholder for li LUT offset (patched later)
}
bool Section::loadSectionFile(const int fontId, const float lineCompression, const bool extraParagraphSpacing,
@@ -217,8 +220,8 @@ bool Section::createSectionFile(const int fontId, const float lineCompression, c
ChapterHtmlSlimParser visitor(
epub, tmpHtmlPath, renderer, fontId, lineCompression, extraParagraphSpacing, paragraphAlignment, viewportWidth,
viewportHeight, hyphenationEnabled,
[this, &lut](std::unique_ptr<Page> page, const uint16_t paragraphIndex) {
lut.push_back({this->onPageComplete(std::move(page)), paragraphIndex});
[this, &lut](std::unique_ptr<Page> page, const uint16_t paragraphIndex, const uint16_t listItemIndex) {
lut.push_back({this->onPageComplete(std::move(page)), paragraphIndex, listItemIndex});
},
embeddedStyle, contentBase, imageBasePath, imageRendering, popupFn, cssParser);
Hyphenator::setPreferredLanguage(epub->getLanguage());
@@ -270,12 +273,18 @@ bool Section::createSectionFile(const int fontId, const float lineCompression, c
serialization::writePod(file, entry.paragraphIndex);
}
// Patch header with final pageCount, lutOffset, anchorMapOffset, and paragraphLutOffset
file.seek(HEADER_SIZE - sizeof(uint32_t) * 3 - sizeof(pageCount));
const uint32_t liLutFileOffset = static_cast<uint32_t>(file.position());
for (const auto& entry : lut) {
serialization::writePod(file, entry.listItemIndex);
}
// Patch header with final pageCount, lutOffset, anchorMapOffset, paragraphLutOffset, and liLutOffset
file.seek(HEADER_SIZE - sizeof(uint32_t) * 4 - sizeof(pageCount));
serialization::writePod(file, pageCount);
serialization::writePod(file, lutOffset);
serialization::writePod(file, anchorMapOffset);
serialization::writePod(file, paragraphLutOffset);
serialization::writePod(file, liLutFileOffset);
// Explicit close() required: member variable persists beyond function scope
file.close();
if (cssParser) {
@@ -289,7 +298,7 @@ std::unique_ptr<Page> Section::loadPageFromSectionFile() {
return nullptr;
}
file.seek(HEADER_SIZE - sizeof(uint32_t) * 3);
file.seek(HEADER_SIZE - sizeof(uint32_t) * 4);
uint32_t lutOffset;
serialization::readPod(file, lutOffset);
file.seek(lutOffset + sizeof(uint32_t) * currentPage);
@@ -310,7 +319,7 @@ std::optional<uint16_t> Section::getPageForAnchor(const std::string& anchor) con
}
const uint32_t fileSize = f.size();
f.seek(HEADER_SIZE - sizeof(uint32_t) * 2);
f.seek(HEADER_SIZE - sizeof(uint32_t) * 3);
uint32_t anchorMapOffset;
serialization::readPod(f, anchorMapOffset);
if (anchorMapOffset == 0 || anchorMapOffset >= fileSize) {
@@ -340,7 +349,7 @@ std::optional<uint16_t> Section::getPageForParagraphIndex(const uint16_t pIndex)
}
const uint32_t fileSize = f.size();
f.seek(HEADER_SIZE - sizeof(uint32_t));
f.seek(HEADER_SIZE - sizeof(uint32_t) * 2);
uint32_t paragraphLutOffset;
serialization::readPod(f, paragraphLutOffset);
if (paragraphLutOffset == 0 || paragraphLutOffset >= fileSize) {
@@ -379,7 +388,7 @@ std::optional<uint16_t> Section::getParagraphIndexForPage(const uint16_t page) c
}
const uint32_t fileSize = f.size();
f.seek(HEADER_SIZE - sizeof(uint32_t));
f.seek(HEADER_SIZE - sizeof(uint32_t) * 2);
uint32_t paragraphLutOffset;
serialization::readPod(f, paragraphLutOffset);
if (paragraphLutOffset == 0 || paragraphLutOffset >= fileSize) {
@@ -403,3 +412,51 @@ std::optional<uint16_t> Section::getParagraphIndexForPage(const uint16_t page) c
serialization::readPod(f, pIdx);
return pIdx;
}
std::optional<uint16_t> Section::getPageForListItemIndex(const uint16_t liIndex) const {
FsFile f;
if (!Storage.openFileForRead("SCT", filePath, f)) {
return std::nullopt;
}
const uint32_t fileSize = f.size();
f.seek(HEADER_SIZE - sizeof(uint32_t));
uint32_t liLutOffset;
serialization::readPod(f, liLutOffset);
if (liLutOffset == 0 || liLutOffset >= fileSize) {
return std::nullopt;
}
// The li LUT shares count with the paragraph LUT; read count from paragraphLutOffset
f.seek(HEADER_SIZE - sizeof(uint32_t) * 2);
uint32_t paragraphLutOffset;
serialization::readPod(f, paragraphLutOffset);
if (paragraphLutOffset == 0 || paragraphLutOffset >= fileSize) {
return std::nullopt;
}
f.seek(paragraphLutOffset);
uint16_t count;
serialization::readPod(f, count);
if (count == 0) {
return std::nullopt;
}
const uint32_t lutEnd = liLutOffset + count * sizeof(uint16_t);
if (lutEnd > fileSize) {
return std::nullopt;
}
f.seek(liLutOffset);
uint16_t resultPage = count - 1;
for (uint16_t i = 0; i < count; i++) {
uint16_t pageLiIdx;
serialization::readPod(f, pageLiIdx);
if (pageLiIdx >= liIndex) {
resultPage = i;
break;
}
}
return resultPage;
}
+3
View File
@@ -46,6 +46,9 @@ class Section {
// Look up the page number for a synthetic paragraph index from XPath p[N].
std::optional<uint16_t> getPageForParagraphIndex(uint16_t pIndex) const;
// Look up the page number for a running list-item index from the li LUT.
std::optional<uint16_t> getPageForListItemIndex(uint16_t liIndex) const;
// Look up the synthetic paragraph index for the given rendered page.
std::optional<uint16_t> getParagraphIndexForPage(uint16_t page) const;
};
@@ -157,6 +157,9 @@ void XMLCALL ChapterHtmlSlimParser::startElement(void* userData, const XML_Char*
if (strcmp(name, "p") == 0) {
self->xpathParagraphIndex++;
}
if (strcmp(name, "li") == 0) {
self->xpathListItemIndex++;
}
// Extract class, style, and id attributes
std::string classAttr;
@@ -449,7 +452,8 @@ void XMLCALL ChapterHtmlSlimParser::startElement(void* userData, const XML_Char*
if (self->currentPage && !self->currentPage->elements.empty() &&
(self->currentPageNextY + imageMarginTop + displayHeight + imageMarginBottom >
self->viewportHeight)) {
self->completePageFn(std::move(self->currentPage), self->xpathParagraphIndex);
self->completePageFn(std::move(self->currentPage), self->xpathParagraphIndex,
self->xpathListItemIndex);
self->completedPageCount++;
self->currentPage.reset(new Page());
if (!self->currentPage) {
@@ -1115,7 +1119,7 @@ bool ChapterHtmlSlimParser::parseAndBuildPages() {
anchorData.push_back({std::move(pendingAnchorId), static_cast<uint16_t>(completedPageCount)});
pendingAnchorId.clear();
}
completePageFn(std::move(currentPage), xpathParagraphIndex);
completePageFn(std::move(currentPage), xpathParagraphIndex, xpathListItemIndex);
completedPageCount++;
currentPage.reset();
currentTextBlock.reset();
@@ -1133,7 +1137,7 @@ void ChapterHtmlSlimParser::addLineToPage(std::shared_ptr<TextBlock> line) {
}
if (currentPageNextY + lineHeight > viewportHeight) {
completePageFn(std::move(currentPage), xpathParagraphIndex);
completePageFn(std::move(currentPage), xpathParagraphIndex, xpathListItemIndex);
completedPageCount++;
currentPage.reset(new Page());
currentPageNextY = 0;
@@ -25,7 +25,7 @@ class ChapterHtmlSlimParser {
std::shared_ptr<Epub> epub;
const std::string& filepath;
GfxRenderer& renderer;
std::function<void(std::unique_ptr<Page>, uint16_t)> completePageFn;
std::function<void(std::unique_ptr<Page>, uint16_t, uint16_t)> completePageFn;
std::function<void()> popupFn; // Popup callback
int depth = 0;
int skipUntilDepth = INT_MAX;
@@ -76,6 +76,7 @@ class ChapterHtmlSlimParser {
std::vector<std::pair<std::string, uint16_t>> anchorData;
std::string pendingAnchorId; // deferred until after previous text block is flushed
uint16_t xpathParagraphIndex = 0;
uint16_t xpathListItemIndex = 0;
// Footnote link tracking
bool insideFootnoteLink = false;
@@ -100,7 +101,7 @@ class ChapterHtmlSlimParser {
const int fontId, const float lineCompression, const bool extraParagraphSpacing,
const uint8_t paragraphAlignment, const uint16_t viewportWidth,
const uint16_t viewportHeight, const bool hyphenationEnabled,
const std::function<void(std::unique_ptr<Page>, uint16_t)>& completePageFn,
const std::function<void(std::unique_ptr<Page>, uint16_t, uint16_t)>& completePageFn,
const bool embeddedStyle, const std::string& contentBase,
const std::string& imageBasePath, const uint8_t imageRendering = 0,
const std::function<void()>& popupFn = nullptr, const CssParser* cssParser = nullptr)
+43 -11
View File
@@ -49,13 +49,14 @@ struct PathSegment {
int index;
};
std::string buildParagraphXPath(const int spineIndex, const std::vector<PathSegment>& path, const int charOffset) {
std::string buildParagraphXPath(const int spineIndex, const std::vector<PathSegment>& path, const int textNodeIndex,
const size_t charOffset) {
std::string xpath = "/body/DocFragment[" + std::to_string(spineIndex + 1) + "]/body";
for (const auto& segment : path) {
xpath += "/" + segment.name + "[" + std::to_string(segment.index) + "]";
}
if (charOffset > 0) {
xpath += "/text()." + std::to_string(charOffset);
if (textNodeIndex > 0 && charOffset > 0) {
xpath += "/text()[" + std::to_string(textNodeIndex) + "]." + std::to_string(charOffset);
}
return xpath;
}
@@ -280,7 +281,7 @@ class XPathParagraphResolver final : public Print {
if (name == "p") {
paragraphCount++;
if (paragraphCount == targetParagraph) {
xpath = buildParagraphXPath(spineIndex, path, 0);
xpath = buildParagraphXPath(spineIndex, path, 0, 0);
stopped = true;
XML_StopParser(parser, XML_FALSE);
}
@@ -410,10 +411,14 @@ class XPathProgressResolver final : public Print {
const int siblingIndex = parentStates.back().nextIndex(name);
path.push_back({name, siblingIndex});
parentStates.emplace_back();
textNodeIndexStack.push_back(0);
pendingTextNode = true;
if (name == "p") {
paragraphDepth++;
paragraphVisibleChars = 0;
}
if (name == "li") {
liDepth++;
}
depth++;
@@ -431,14 +436,23 @@ class XPathProgressResolver final : public Print {
insideBody = false;
parentStates.clear();
path.clear();
textNodeIndexStack.clear();
return;
}
if (name == "p" && paragraphDepth > 0) {
paragraphDepth--;
paragraphVisibleChars = 0;
}
if (name == "li" && liDepth > 0) {
liDepth--;
}
if (!textNodeIndexStack.empty()) {
textNodeIndexStack.pop_back();
}
if (paragraphDepth > 0 || liDepth > 0) {
pendingTextNode = true;
}
if (!path.empty()) {
path.pop_back();
}
@@ -448,23 +462,38 @@ class XPathProgressResolver final : public Print {
}
void onCharacterData(const XML_Char* data, const int len) {
if (!insideBody || paragraphDepth <= 0 || len <= 0 || stopped) {
if (!insideBody || (paragraphDepth <= 0 && liDepth <= 0) || len <= 0 || stopped) {
return;
}
const size_t codepointCount = countUtf8Codepoints(data, len);
if (codepointCount == 0) {
return;
}
// Start a new text node on first non-empty content after any element boundary.
// Only counting non-empty nodes matches KOReader's text()[N] indexing behavior,
// which skips empty text nodes created by bare <a id="anchor"/> anchors.
if (pendingTextNode) {
if (!textNodeIndexStack.empty()) {
textNodeIndexStack.back()++;
}
textNodeStartChars = visibleChars;
pendingTextNode = false;
}
const size_t nextVisibleChars = visibleChars + codepointCount;
if (targetVisibleChar <= nextVisibleChars) {
const size_t delta = targetVisibleChar - visibleChars;
const int charOffset = static_cast<int>(paragraphVisibleChars + delta);
xpath = buildParagraphXPath(spineIndex, path, std::max(1, charOffset));
const int texNode = textNodeIndexStack.empty() ? 0 : textNodeIndexStack.back();
const size_t charOff = visibleChars - textNodeStartChars + delta;
xpath = buildParagraphXPath(spineIndex, path, texNode, charOff);
stopped = true;
XML_StopParser(parser, XML_FALSE);
return;
}
visibleChars = nextVisibleChars;
paragraphVisibleChars += codepointCount;
}
XML_Parser parser = nullptr;
@@ -472,11 +501,14 @@ class XPathProgressResolver final : public Print {
bool parseOk = true;
bool insideBody = false;
bool stopped = false;
bool pendingTextNode = true;
int depth = 0;
int bodyDepth = -1;
int paragraphDepth = 0;
int liDepth = 0;
size_t visibleChars = 0;
size_t paragraphVisibleChars = 0;
size_t textNodeStartChars = 0;
std::vector<int> textNodeIndexStack;
std::vector<ParentState> parentStates;
std::vector<PathSegment> path;
std::string xpath;
+407 -68
View File
@@ -39,58 +39,154 @@ int parseCharOffset(const std::string& xpath) {
return val;
}
// Parse the N from text()[N] in the XPath (1-based; defaults to 1 if absent or 1).
int parseTextNodeIndex(const std::string& xpath) {
const size_t textPos = xpath.rfind("text()[");
if (textPos == std::string::npos) return 1;
const size_t numStart = textPos + 7; // strlen("text()[")
const size_t numEnd = xpath.find(']', numStart);
if (numEnd == std::string::npos || numEnd == numStart) return 1;
int val = 0;
for (size_t i = numStart; i < numEnd; i++) {
if (xpath[i] < '0' || xpath[i] > '9') return 1;
val = val * 10 + (xpath[i] - '0');
}
return val > 0 ? val : 1;
}
// Parsed representation of one step in the XPath ancestry.
struct XPathStep {
char tag[12]; // element name, null-terminated
int siblingIndex; // 1-based sibling index, or 0 if unspecified (treat as 1)
};
static constexpr int MAX_XPATH_DEPTH = 16;
// Parse the XPath segment between /body/DocFragment[N]/body/ and text()[N].offset
// into an ordered sequence of steps. Returns step count, 0 on failure.
// Example input: "/body/DocFragment[1]/body/div[1]/ul/li[4]/text()[1].51"
// Fills steps with: {div,1}, {ul,1}, {li,4}
int parseXPathSteps(const std::string& xpath, XPathStep steps[MAX_XPATH_DEPTH]) {
static const char kBodyFrag[] = "/body/DocFragment[";
const size_t fragPos = xpath.find(kBodyFrag);
if (fragPos == std::string::npos) return 0;
const size_t afterBracket = xpath.find(']', fragPos + strlen(kBodyFrag));
if (afterBracket == std::string::npos) return 0;
static const char kBody[] = "/body/";
if (xpath.compare(afterBracket + 1, strlen(kBody), kBody) != 0) return 0;
size_t pos = afterBracket + 1 + strlen(kBody);
const size_t textPos = xpath.rfind("/text()");
if (textPos == std::string::npos || textPos <= pos) return 0;
int count = 0;
while (pos < textPos && count < MAX_XPATH_DEPTH) {
const size_t slash = xpath.find('/', pos);
const size_t segEnd = (slash < textPos) ? slash : textPos;
XPathStep& step = steps[count];
const size_t bracket = xpath.find('[', pos);
const size_t nameEnd = (bracket != std::string::npos && bracket < segEnd) ? bracket : segEnd;
const size_t nameLen = nameEnd - pos;
if (nameLen == 0 || nameLen >= sizeof(step.tag)) return 0;
memcpy(step.tag, xpath.c_str() + pos, nameLen);
step.tag[nameLen] = '\0';
if (bracket != std::string::npos && bracket < segEnd) {
const size_t closeBracket = xpath.find(']', bracket + 1);
if (closeBracket == std::string::npos || closeBracket > segEnd) return 0;
int idx = 0;
for (size_t i = bracket + 1; i < closeBracket; i++) {
if (xpath[i] < '0' || xpath[i] > '9') return 0;
idx = idx * 10 + (xpath[i] - '0');
}
step.siblingIndex = idx;
} else {
step.siblingIndex = 1;
}
count++;
pos = (slash < textPos) ? slash + 1 : textPos;
}
return count;
}
class ParagraphStreamer final : public Print {
size_t bytesWritten = 0;
bool globalInTag = false;
bool globalInEntity = false;
enum { IDLE, SAW_LT, SAW_LT_P } pState = IDLE;
static constexpr size_t MAX_ENTITY_SIZE = 16;
char entityBuffer[MAX_ENTITY_SIZE] = {};
size_t entityLen = 0;
// Forward mode: count paragraphs at a byte offset
// Forward mode: count <p> paragraphs at a byte offset (legacy, used by generateXPath)
size_t fwdTarget;
int fwdResult = 0;
bool fwdCaptured = false;
// Reverse mode: find position of Nth paragraph + char offset
int revParagraph;
// Reverse mode shared state
int revChar;
int pCount = 0;
bool revPFound = false;
bool revDone = false;
int revVisChars = 0; // Visible chars counted WITHIN target paragraph
size_t totalVisChars = 0; // Total visible chars in entire file
size_t targetVisChars = 0; // Visible chars from start of file to target position
int revVisChars = 0;
size_t totalVisChars = 0;
size_t targetVisChars = 0;
void onP() {
pCount++;
if (!revPFound && revParagraph > 0 && pCount >= revParagraph) {
revPFound = true;
revVisChars = 0;
if (revChar <= 0) {
targetVisChars = totalVisChars;
revDone = true;
}
}
}
// --- Legacy reverse mode (paragraph index only, no ancestry) ---
int revParagraph = 0;
int pCount = 0;
int paragraphAtMatch = 0;
int liCount = 0;
int liCountAtMatch = 0;
int targetTextNode = 1;
int currentTextNode = 0;
int paragraphHtmlDepth = -1;
// --- Ancestry-aware reverse mode ---
const XPathStep* steps = nullptr;
int stepCount = 0;
int siblingCounters[MAX_XPATH_DEPTH] = {};
bool insideStep[MAX_XPATH_DEPTH] = {};
int htmlDepth = 0;
int stepEnteredAtDepth[MAX_XPATH_DEPTH] = {};
// Tag name accumulation
enum TagParseState { TAG_IDLE, TAG_IN_NAME, TAG_ATTRS } tagState = TAG_IDLE;
bool tagIsClose = false;
char tagName[12] = {};
int tagNameLen = 0;
int matchedDepth = 0;
// Anchor ID capture
static constexpr int MAX_ANCHOR_ID = 64;
char capturedAnchorId[MAX_ANCHOR_ID] = {};
int capturedAnchorIdLen = 0;
bool capturingAnchorTag = false;
enum IdScanState { ID_SCAN, ID_I, ID_D, ID_EQ, ID_IN_VALUE_D, ID_IN_VALUE_S } idState = ID_SCAN;
bool inAttrQuote =
false; // true while inside a quoted attribute value (prevents '/' from being treated as self-close)
char attrQuoteChar = 0;
void onVisibleCodepoint() {
totalVisChars++;
if (revPFound && !revDone) {
revVisChars++;
if (revVisChars >= revChar) {
targetVisChars = totalVisChars;
revDone = true;
// Ancestry mode: count only while inside the fully-matched element and in the target text node.
// Legacy mode: count only while still inside the matched paragraph and in the target text node.
const bool inTargetNode = (stepCount > 0) ? (matchedDepth == stepCount && currentTextNode == targetTextNode)
: (paragraphHtmlDepth >= 0 && currentTextNode == targetTextNode);
if (inTargetNode) {
revVisChars++;
if (revVisChars >= revChar) {
targetVisChars = totalVisChars;
revDone = true;
}
}
}
}
void onVisibleText(const char* text) {
if (!text) {
return;
}
if (!text) return;
const unsigned char* ptr = reinterpret_cast<const unsigned char*>(text);
while (*ptr != 0) {
utf8NextCodepoint(&ptr);
@@ -99,26 +195,229 @@ class ParagraphStreamer final : public Print {
}
void flushEntityAsLiteral() {
for (size_t i = 0; i < entityLen; i++) {
onVisibleCodepoint();
}
for (size_t i = 0; i < entityLen; i++) onVisibleCodepoint();
}
void finishEntity() {
entityBuffer[entityLen] = '\0';
const char* resolved = lookupHtmlEntity(entityBuffer, entityLen);
if (resolved) {
if (resolved)
onVisibleText(resolved);
} else {
else
flushEntityAsLiteral();
}
globalInEntity = false;
entityLen = 0;
}
void onLegacyP() {
pCount++;
if (!revPFound && revParagraph > 0 && pCount >= revParagraph) {
revPFound = true;
revVisChars = 0;
paragraphHtmlDepth = htmlDepth;
currentTextNode = 1;
if (revChar <= 0 && targetTextNode <= 1) {
targetVisChars = totalVisChars;
revDone = true;
}
}
}
void onOpenTag() {
htmlDepth++;
if (stepCount == 0) {
if (strcasecmp(tagName, "p") == 0) onLegacyP();
return;
}
// Capture <a id> inside the fully-matched element even after target char is found
if (revPFound && matchedDepth == stepCount && capturedAnchorIdLen == 0 && strcasecmp(tagName, "a") == 0) {
capturingAnchorTag = true;
idState = ID_SCAN;
}
if (revDone) return;
if (strcasecmp(tagName, "p") == 0) pCount++;
if (strcasecmp(tagName, "li") == 0) liCount++;
if (matchedDepth < stepCount) {
const XPathStep& target = steps[matchedDepth];
if (strcasecmp(tagName, target.tag) == 0) {
// Count only direct children of the previously matched ancestor step.
// For step 0 any depth is valid; subsequent steps must be exactly one level deeper.
const bool atCorrectDepth = (matchedDepth == 0) || (htmlDepth == stepEnteredAtDepth[matchedDepth - 1] + 1);
if (!atCorrectDepth) return;
siblingCounters[matchedDepth]++;
if (siblingCounters[matchedDepth] == target.siblingIndex) {
insideStep[matchedDepth] = true;
stepEnteredAtDepth[matchedDepth] = htmlDepth;
matchedDepth++;
if (matchedDepth == stepCount) {
paragraphAtMatch = pCount;
liCountAtMatch = liCount;
revPFound = true;
capturedAnchorIdLen = 0;
revVisChars = 0;
currentTextNode = 1; // Reset text node counter for this element
if (revChar <= 0 && targetTextNode <= 1) {
targetVisChars = totalVisChars;
revDone = true;
}
}
}
}
}
}
void onCloseTag() {
// Legacy mode: each direct child element closing advances the text node index.
if (stepCount == 0 && revPFound && !revDone && paragraphHtmlDepth >= 0 && htmlDepth == paragraphHtmlDepth + 1) {
currentTextNode++;
if (currentTextNode == targetTextNode && revChar <= 0) {
targetVisChars = totalVisChars;
revDone = true;
}
}
// Legacy mode: stop tracking when the matched paragraph itself closes.
if (stepCount == 0 && revPFound && !revDone && paragraphHtmlDepth >= 0 && htmlDepth == paragraphHtmlDepth) {
revPFound = false;
paragraphHtmlDepth = -1;
}
// Ancestry mode: advance text node when a direct child of the fully-matched element closes.
if (stepCount > 0 && matchedDepth == stepCount && revPFound && !revDone) {
const int elementDepth = stepEnteredAtDepth[stepCount - 1];
if (htmlDepth == elementDepth + 1) {
currentTextNode++;
if (currentTextNode == targetTextNode && revChar <= 0) {
targetVisChars = totalVisChars;
revDone = true;
}
}
}
if (stepCount > 0 && matchedDepth > 0) {
const int step = matchedDepth - 1;
if (insideStep[step] && htmlDepth == stepEnteredAtDepth[step]) {
insideStep[step] = false;
matchedDepth--;
// If the fully-matched element just closed without finding the target, abort.
if (matchedDepth < stepCount && revPFound && !revDone) {
revPFound = false;
}
for (int i = matchedDepth + 1; i < stepCount; i++) {
siblingCounters[i] = 0;
insideStep[i] = false;
stepEnteredAtDepth[i] = -1;
}
}
}
if (htmlDepth > 0) htmlDepth--;
}
void processByteInTag(uint8_t c) {
switch (tagState) {
case TAG_IDLE:
if (c == '/') {
tagIsClose = true;
tagState = TAG_IN_NAME;
} else if (c != '!' && c != '?') {
tagIsClose = false;
tagName[0] = static_cast<char>(c);
tagNameLen = 1;
tagState = TAG_IN_NAME;
}
break;
case TAG_IN_NAME:
if (c == '>' || c == ' ' || c == '\t' || c == '\n' || c == '\r' || c == '/') {
tagName[tagNameLen] = '\0';
if (tagNameLen > 0) {
if (tagIsClose)
onCloseTag();
else
onOpenTag();
// Self-closing open tag (<br/>). Don't double-fire for close tags (</br/>).
if (c == '/' && !tagIsClose) onCloseTag();
}
tagNameLen = 0;
tagState = (c == '>') ? TAG_IDLE : TAG_ATTRS;
} else if (tagNameLen + 1 < static_cast<int>(sizeof(tagName))) {
tagName[tagNameLen++] = static_cast<char>(c);
}
break;
case TAG_ATTRS:
// Track quoted attribute values so '/' inside them is not mistaken for self-closing.
if (!inAttrQuote) {
if (c == '"' || c == '\'') {
inAttrQuote = true;
attrQuoteChar = c;
}
} else if (c == attrQuoteChar) {
inAttrQuote = false;
attrQuoteChar = 0;
}
if (capturingAnchorTag) {
switch (idState) {
case ID_SCAN:
idState = (c == 'i' || c == 'I') ? ID_I : ID_SCAN;
break;
case ID_I:
idState = (c == 'd' || c == 'D') ? ID_D : ID_SCAN;
break;
case ID_D:
idState = (c == '=') ? ID_EQ : ID_SCAN;
break;
case ID_EQ:
if (c == '"')
idState = ID_IN_VALUE_D;
else if (c == '\'')
idState = ID_IN_VALUE_S;
break;
case ID_IN_VALUE_D:
if (c == '"') {
capturedAnchorId[capturedAnchorIdLen] = '\0';
capturingAnchorTag = false;
} else if (capturedAnchorIdLen + 1 < MAX_ANCHOR_ID)
capturedAnchorId[capturedAnchorIdLen++] = c;
break;
case ID_IN_VALUE_S:
if (c == '\'') {
capturedAnchorId[capturedAnchorIdLen] = '\0';
capturingAnchorTag = false;
} else if (capturedAnchorIdLen + 1 < MAX_ANCHOR_ID)
capturedAnchorId[capturedAnchorIdLen++] = c;
break;
}
}
// Only treat '/' as self-closing when outside a quoted attribute value.
if (c == '/' && !inAttrQuote) {
onCloseTag();
capturingAnchorTag = false;
}
break;
}
}
public:
explicit ParagraphStreamer(size_t targetByte) : fwdTarget(targetByte), revParagraph(0), revChar(0) {}
ParagraphStreamer(int paragraph, int charOff) : fwdTarget(SIZE_MAX), revParagraph(paragraph), revChar(charOff) {}
explicit ParagraphStreamer(size_t targetByte) : fwdTarget(targetByte), revChar(0) {
memset(stepEnteredAtDepth, -1, sizeof(stepEnteredAtDepth));
}
ParagraphStreamer(int paragraph, int charOff, int textNodeIdx = 1)
: fwdTarget(SIZE_MAX), revChar(charOff), revParagraph(paragraph), targetTextNode(textNodeIdx) {
memset(stepEnteredAtDepth, -1, sizeof(stepEnteredAtDepth));
}
ParagraphStreamer(const XPathStep* xpathSteps, int xpathStepCount, int charOff, int textNodeIdx = 1)
: fwdTarget(SIZE_MAX),
revChar(charOff),
steps(xpathSteps),
stepCount(xpathStepCount),
targetTextNode(textNodeIdx) {
memset(stepEnteredAtDepth, -1, sizeof(stepEnteredAtDepth));
}
size_t write(uint8_t c) override {
if (!fwdCaptured && bytesWritten >= fwdTarget) {
@@ -135,7 +434,6 @@ class ParagraphStreamer final : public Print {
globalInEntity = false;
entityLen = 0;
}
if (globalInEntity) {
if (c == ';') {
finishEntity();
@@ -145,36 +443,42 @@ class ParagraphStreamer final : public Print {
entityLen = 0;
}
}
} else if (c == '<') {
return 1;
}
if (c == '<') {
globalInTag = true;
tagState = TAG_IDLE;
tagNameLen = 0;
tagIsClose = false;
capturingAnchorTag = false;
idState = ID_SCAN;
inAttrQuote = false;
attrQuoteChar = 0;
} else if (c == '>') {
globalInTag = false;
} else if (!globalInTag) {
inAttrQuote = false;
if (tagState == TAG_IN_NAME && tagNameLen > 0) {
tagName[tagNameLen] = '\0';
if (tagIsClose)
onCloseTag();
else
onOpenTag();
tagNameLen = 0;
}
tagState = TAG_IDLE;
} else if (globalInTag) {
processByteInTag(c);
} else {
if (c == '&') {
globalInEntity = true;
entityBuffer[0] = '&';
entityLen = 1;
} else {
const bool startsCodepoint = (c & 0xC0) != 0x80;
if (startsCodepoint) {
onVisibleCodepoint();
}
if (startsCodepoint) onVisibleCodepoint();
}
}
// Paragraph detection
switch (pState) {
case IDLE:
if (c == '<') pState = SAW_LT;
break;
case SAW_LT:
pState = (c == 'p' || c == 'P') ? SAW_LT_P : ((c == '<') ? SAW_LT : IDLE);
break;
case SAW_LT_P:
if (c == '>' || c == '/' || c == ' ' || c == '\t' || c == '\n' || c == '\r') onP();
pState = (c == '<') ? SAW_LT : IDLE;
break;
}
return 1;
}
@@ -183,10 +487,14 @@ class ParagraphStreamer final : public Print {
return size;
}
public:
int paragraphCount() const { return fwdCaptured ? fwdResult : pCount; }
int getParagraphAtMatch() const { return paragraphAtMatch; }
int getListItemAtMatch() const { return liCountAtMatch; }
const char* getCapturedAnchorId() const { return capturedAnchorIdLen > 0 ? capturedAnchorId : nullptr; }
size_t totalBytes() const { return bytesWritten; }
bool found() const { return revDone || revPFound; }
size_t getTotalVisChars() const { return totalVisChars; }
size_t getTargetVisChars() const { return targetVisChars; }
float progress() const {
return totalVisChars > 0 ? static_cast<float>(targetVisChars) / static_cast<float>(totalVisChars) : 0.0f;
}
@@ -200,12 +508,14 @@ bool streamSpine(const std::shared_ptr<Epub>& epub, int spineIndex, ParagraphStr
KOReaderPosition ProgressMapper::toKOReader(const std::shared_ptr<Epub>& epub, const CrossPointPosition& pos) {
KOReaderPosition result;
float intra = (pos.totalPages > 0) ? static_cast<float>(pos.pageNumber) / static_cast<float>(pos.totalPages) : 0.0f;
float intra =
(pos.totalPages > 1) ? static_cast<float>(pos.pageNumber) / static_cast<float>(pos.totalPages - 1) : 0.0f;
result.percentage = epub->calculateProgress(pos.spineIndex, intra);
if (pos.hasParagraphIndex && pos.paragraphIndex > 0) {
// Progress-based XPath correctly handles both <p> and <li> positions.
result.xpath = ChapterXPathResolver::findXPathForProgress(epub, pos.spineIndex, intra);
// Fall back to paragraph-index lookup when progress-based resolution fails.
if (result.xpath.empty() && pos.hasParagraphIndex && pos.paragraphIndex > 0) {
result.xpath = ChapterXPathResolver::findXPathForParagraph(epub, pos.spineIndex, pos.paragraphIndex);
} else {
result.xpath = ChapterXPathResolver::findXPathForProgress(epub, pos.spineIndex, intra);
}
if (result.xpath.empty()) {
result.xpath = generateXPath(epub, pos.spineIndex, intra);
@@ -228,11 +538,13 @@ CrossPointPosition ProgressMapper::toCrossPoint(const std::shared_ptr<Epub>& epu
const int docFrag = parseIndex(koPos.xpath, "/body/DocFragment[");
const int xpathP = parseIndex(koPos.xpath, "/p[", true);
const int xpathChar = parseCharOffset(koPos.xpath);
const int xpathTextNode = parseTextNodeIndex(koPos.xpath);
const int xpathSpine = (docFrag >= 1) ? (docFrag - 1) : -1;
if (xpathP > 0) {
result.paragraphIndex = static_cast<uint16_t>(xpathP);
result.hasParagraphIndex = true;
}
XPathStep xpathSteps[MAX_XPATH_DEPTH];
const int xpathStepCount = parseXPathSteps(koPos.xpath, xpathSteps);
// Use ancestry mode whenever the XPath has a structured path (always more accurate than global counting).
const bool useAncestry = xpathStepCount > 0;
if (xpathSpine >= 0 && xpathSpine < spineCount) {
result.spineIndex = xpathSpine;
@@ -261,11 +573,37 @@ CrossPointPosition ProgressMapper::toCrossPoint(const std::shared_ptr<Epub>& epu
if (spineSize == 0 || result.totalPages == 0) return result;
float intra = 0.0f;
if (xpathP > 0) {
ParagraphStreamer s(xpathP, xpathChar);
if (useAncestry) {
ParagraphStreamer s(xpathSteps, xpathStepCount, xpathChar, xpathTextNode);
if (streamSpine(epub, result.spineIndex, s) && s.found()) {
intra = s.progress();
LOG_DBG("PM", "XPath p[%d]+%d -> %.1f%%", xpathP, xpathChar, intra * 100);
const int pAtMatch = s.getParagraphAtMatch();
if (pAtMatch > 0) {
result.paragraphIndex = static_cast<uint16_t>(pAtMatch);
result.hasParagraphIndex = true;
}
if (xpathStepCount > 0 && strcasecmp(xpathSteps[xpathStepCount - 1].tag, "li") == 0) {
const int liAtMatch = s.getListItemAtMatch();
if (liAtMatch > 0) {
result.liIndex = static_cast<uint16_t>(liAtMatch);
result.hasLiIndex = true;
}
}
const char* anchorId = s.getCapturedAnchorId();
if (anchorId) {
strncpy(result.xpathAnchorId, anchorId, sizeof(result.xpathAnchorId) - 1);
}
LOG_DBG("PM", "XPath ancestry(%s[%d])/text()[%d]+%d -> %.1f%% (target=%zu total=%zu p~%d li~%d anchor=%s)",
xpathSteps[xpathStepCount - 1].tag, xpathSteps[xpathStepCount - 1].siblingIndex, xpathTextNode, xpathChar,
intra * 100, s.getTargetVisChars(), s.getTotalVisChars(), pAtMatch,
result.hasLiIndex ? static_cast<int>(result.liIndex) : 0, anchorId ? anchorId : "none");
}
} else if (xpathP > 0) {
ParagraphStreamer s(xpathP, xpathChar, xpathTextNode);
if (streamSpine(epub, result.spineIndex, s) && s.found()) {
intra = s.progress();
LOG_DBG("PM", "XPath p[%d]/text()[%d]+%d -> %.1f%% (target=%zu total=%zu)", xpathP, xpathTextNode, xpathChar,
intra * 100, s.getTargetVisChars(), s.getTotalVisChars());
}
}
if (intra <= 0.0f) {
@@ -273,7 +611,8 @@ CrossPointPosition ProgressMapper::toCrossPoint(const std::shared_ptr<Epub>& epu
intra = std::max(0.0f, std::min(1.0f, static_cast<float>(bytesIn) / static_cast<float>(spineSize)));
}
result.pageNumber = std::max(0, std::min(static_cast<int>(intra * result.totalPages), result.totalPages - 1));
result.pageNumber = std::max(
0, std::min(static_cast<int>(intra * static_cast<float>(result.totalPages - 1) + 0.5f), result.totalPages - 1));
LOG_DBG("PM", "<- KO: %.2f%% %s -> spine=%d page=%d/%d", koPos.percentage * 100, koPos.xpath.c_str(),
result.spineIndex, result.pageNumber, result.totalPages);
return result;
+3
View File
@@ -13,6 +13,9 @@ struct CrossPointPosition {
int totalPages; // Total pages in the current spine item
uint16_t paragraphIndex = 0; // 1-based synthetic paragraph index from XPath p[N]
bool hasParagraphIndex = false; // True when paragraphIndex was resolved from XPath
uint16_t liIndex = 0; // Running <li> count at the matched XPath element
bool hasLiIndex = false; // True when target element is <li> and liIndex was resolved
char xpathAnchorId[64] = {}; // First <a id> captured inside the matched XPath element
};
/**
+51 -8
View File
@@ -7,6 +7,7 @@
#include <WiFi.h>
#include <esp_sntp.h>
#include <algorithm>
#include <cassert>
#include "Epub/Section.h"
@@ -183,15 +184,57 @@ void KOReaderSyncActivity::performSync() {
KOReaderPosition koPos = {remoteProgress.progress, remoteProgress.percentage};
remotePosition = ProgressMapper::toCrossPoint(epub, koPos, currentSpineIndex, totalPagesInSpine);
// If XPath carried a paragraph index, refine the page using the section cache's
// per-page paragraph LUT instead of anchor matching.
if (remotePosition.hasParagraphIndex) {
// Refine page using section cache LUTs: li index, anchor, or paragraph index.
if (remotePosition.hasLiIndex || remotePosition.xpathAnchorId[0] != '\0' || remotePosition.hasParagraphIndex) {
Section tempSection(epub, remotePosition.spineIndex, renderer);
const auto paragraphPage = tempSection.getPageForParagraphIndex(remotePosition.paragraphIndex);
if (paragraphPage.has_value()) {
LOG_DBG("KOSync", "Paragraph %u resolved to page %d (was %d)", remotePosition.paragraphIndex, *paragraphPage,
remotePosition.pageNumber);
remotePosition.pageNumber = *paragraphPage;
bool refined = false;
if (remotePosition.hasLiIndex) {
const auto liPage = tempSection.getPageForListItemIndex(remotePosition.liIndex);
if (liPage.has_value()) {
LOG_DBG("KOSync", "Li index %u -> page %d (was %d)", remotePosition.liIndex, *liPage,
remotePosition.pageNumber);
remotePosition.pageNumber = *liPage;
refined = true;
} else {
LOG_DBG("KOSync", "Li index %u not found in section LUT", remotePosition.liIndex);
}
}
if (!refined && remotePosition.xpathAnchorId[0] != '\0') {
const auto anchorPage = tempSection.getPageForAnchor(std::string(remotePosition.xpathAnchorId));
if (anchorPage.has_value()) {
LOG_DBG("KOSync", "Anchor '%s' -> page %d (was %d)", remotePosition.xpathAnchorId, *anchorPage,
remotePosition.pageNumber);
remotePosition.pageNumber = *anchorPage;
refined = true;
} else {
LOG_DBG("KOSync", "Anchor '%s' not found in section cache", remotePosition.xpathAnchorId);
}
}
if (!refined && remotePosition.hasParagraphIndex) {
const auto paragraphPage = tempSection.getPageForParagraphIndex(remotePosition.paragraphIndex);
const auto nextParagraphPage = tempSection.getPageForParagraphIndex(remotePosition.paragraphIndex + 1);
if (paragraphPage.has_value()) {
int refinedPage = std::max(remotePosition.pageNumber, static_cast<int>(*paragraphPage));
if (nextParagraphPage.has_value()) {
const int lutSpan = static_cast<int>(*nextParagraphPage) - static_cast<int>(*paragraphPage);
// Only cap when the LUT span is >1. A span of 1 means the LUT granularity is too
// coarse to trust over the intra-spine position (e.g. a stale cache where the paragraph
// occupies different pages than at build time).
if (lutSpan > 1 && refinedPage >= static_cast<int>(*nextParagraphPage)) {
refinedPage = static_cast<int>(*nextParagraphPage) - 1;
}
}
char nextParaBuf[8];
if (nextParagraphPage.has_value())
snprintf(nextParaBuf, sizeof(nextParaBuf), "%d", *nextParagraphPage);
else
snprintf(nextParaBuf, sizeof(nextParaBuf), "none");
LOG_DBG("KOSync", "Paragraph %u -> LUT page %d, nextPara page %s, intra page %d, using %d",
remotePosition.paragraphIndex, *paragraphPage, nextParaBuf, remotePosition.pageNumber, refinedPage);
remotePosition.pageNumber = refinedPage;
} else {
LOG_DBG("KOSync", "Paragraph %u not found in section LUT", remotePosition.paragraphIndex);
}
}
}
// localProgress was pre-computed in EpubReaderActivity before the Epub was released.