eric/Crosspoint
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: dict based Hyphenation (#305)

Browse Source 
## Summary

* Adds (optional) Hyphenation for English, French, German, Russian
languages

## Additional Context

* Included hyphenation dictionaries add approximately 280kb to the flash
usage (German alone takes 200kb)
* Trie encoded dictionaries are adopted from hypher project
(https://github.com/typst/hypher)
* Soft hyphens (and other explicit hyphens) take precedence over
dict-based hyphenation. Overall, the hyphenation rules are quite
aggressive, as I believe it makes more sense on our smaller screen.

---------

Co-authored-by: Dave Allie <dave@daveallie.com>
This commit is contained in:
Arthur Tazhitdinov
2026-01-19 12:56:26 +00:00
committed by GitHub
co-authored by Dave Allie
parent 0beaaaacba
commit 09f8fa84b9
40 changed files with 36465 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files
lib/Epub/Epub/parsers/ChapterHtmlSlimParser.cpp
+1 -16
View File
@@ -51,7 +51,7 @@ void ChapterHtmlSlimParser::startNewTextBlock(const TextBlock::Style style) {
makePages();
}
currentTextBlock.reset(new ParsedText(style, extraParagraphSpacing));
currentTextBlock.reset(new ParsedText(style, extraParagraphSpacing, hyphenationEnabled));
}
void XMLCALL ChapterHtmlSlimParser::startElement(void* userData, const XML_Char* name, const XML_Char** atts) {
@@ -170,21 +170,6 @@ void XMLCALL ChapterHtmlSlimParser::characterData(void* userData, const XML_Char
continue;
}
// Skip soft-hyphen with UTF-8 representation (U+00AD) = 0xC2 0xAD
const XML_Char SHY_BYTE_1 = static_cast<XML_Char>(0xC2);
const XML_Char SHY_BYTE_2 = static_cast<XML_Char>(0xAD);
// 1. Check for the start of the 2-byte Soft Hyphen sequence
if (s[i] == SHY_BYTE_1) {
// 2. Check if the next byte exists AND if it completes the sequence
// We must check i + 1 < len to prevent reading past the end of the buffer.
if ((i + 1 < len) && (s[i + 1] == SHY_BYTE_2)) {
// Sequence 0xC2 0xAD found!
// Skip the current byte (0xC2) and the next byte (0xAD)
i++; // Increment 'i' one more time to skip the 0xAD byte
continue; // Skip the rest of the loop and move to the next iteration
}
}
// Skip Zero Width No-Break Space / BOM (U+FEFF) = 0xEF 0xBB 0xBF
const XML_Char FEFF_BYTE_1 = static_cast<XML_Char>(0xEF);
const XML_Char FEFF_BYTE_2 = static_cast<XML_Char>(0xBB);