eric/Crosspoint
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: Fix hyphenation and rendering of decomposed characters (#1037)

Browse Source 
## Summary

* This PR fixes decomposed diacritic handling end-to-end:
- Hyphenation: normalize common Latin base+combining sequences to
precomposed codepoints before Liang pattern matching, so decomposed
words hyphenate correctly
- Rendering: correct combining-mark placement logic so non-spacing marks
are attached to the preceding base glyph in normal and rotated text
rendering paths, with corresponding text-bounds consistency updates.
- Hyphenation around non breaking space variants have been fixed (and
extended)
- Hyphenation of terms that already included of hyphens were fixed to
include Liang pattern application (eg "US-Satellitensystem" was
*exclusively* broken at the existing hyphen)

## Additional Context

* Before
<img width="800" height="480" alt="2"
src="https://github.com/user-attachments/assets/b9c515c4-ab75-45cc-8b52-f4d86bce519d"
/>


* After
<img width="480" height="800" alt="fix1"
src="https://github.com/user-attachments/assets/4999f6a8-f51c-4c0a-b144-f153f77ddb57"
/>
<img width="800" height="480" alt="fix2"
src="https://github.com/user-attachments/assets/7355126b-80c7-441f-b390-4e0897ee3fb6"
/>

* Note 1: the hyphenation fix is not a 100% bullet proof implementation.
It adds composition of *common* base+combining sequences (e.g. O +
U+0308 -> Ö) during codepoint collection. A complete solution would
require implementing proper Unicode normalization (at least NFC,
possibly NFKC in specific cases) before hyphenation and rendering,
instead of hand-mapping a few combining marks. That was beyond the scope
of this fix.

* Note 2: the render fix should be universal and not limited to the
constraints outlined above: it properly x-centers the compund glyph over
the previous one, and it uses at least 1pt of visual distance in y.

Before:
<img width="478" height="167" alt="Image"
src="https://github.com/user-attachments/assets/f8db60d5-35b1-4477-96d0-5003b4e4a2a1"
/>

After: 
<img width="479" height="180" alt="Image"
src="https://github.com/user-attachments/assets/1b48ef97-3a77-475a-8522-23f4aca8e904"
/>

* This should resolve the issues described in #998 
---

### AI Usage

While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.

Did you use AI tools to help write this code? _**PARTIALLY**_
This commit is contained in:
jpirnay
2026-02-22 13:11:07 +11:00
committed by GitHub
parent 472c56284e
commit c0ed60daa6
8 changed files with 459 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
lib/Epub/Epub/hyphenation/HyphenationCommon.cpp
+207
View File
@@ -174,6 +174,213 @@ std::vector<CodepointInfo> collectCodepoints(const std::string& word) {
while (*ptr != 0) {
const unsigned char* current = ptr;
const uint32_t cp = utf8NextCodepoint(&ptr);
// If this is a combining diacritic (e.g., U+0301 = acute) and there's
// a previous base character that can be composed into a single
// precomposed Unicode scalar (Latin-1 / Latin-Extended), do that
// composition here. This provides lightweight NFC-like behavior for
// common Western European diacritics (acute, grave, circumflex, tilde,
// diaeresis, cedilla) without pulling in a full Unicode normalization
// library.
if (!cps.empty()) {
uint32_t prev = cps.back().value;
uint32_t composed = 0;
switch (cp) {
case 0x0300: // grave
switch (prev) {
case 0x0041:
composed = 0x00C0;
break; // A -> À
case 0x0061:
composed = 0x00E0;
break; // a -> à
case 0x0045:
composed = 0x00C8;
break; // E -> È
case 0x0065:
composed = 0x00E8;
break; // e -> è
case 0x0049:
composed = 0x00CC;
break; // I -> Ì
case 0x0069:
composed = 0x00EC;
break; // i -> ì
case 0x004F:
composed = 0x00D2;
break; // O -> Ò
case 0x006F:
composed = 0x00F2;
break; // o -> ò
case 0x0055:
composed = 0x00D9;
break; // U -> Ù
case 0x0075:
composed = 0x00F9;
break; // u -> ù
default:
break;
}
break;
case 0x0301: // acute
switch (prev) {
case 0x0041:
composed = 0x00C1;
break; // A -> Á
case 0x0061:
composed = 0x00E1;
break; // a -> á
case 0x0045:
composed = 0x00C9;
break; // E -> É
case 0x0065:
composed = 0x00E9;
break; // e -> é
case 0x0049:
composed = 0x00CD;
break; // I -> Í
case 0x0069:
composed = 0x00ED;
break; // i -> í
case 0x004F:
composed = 0x00D3;
break; // O -> Ó
case 0x006F:
composed = 0x00F3;
break; // o -> ó
case 0x0055:
composed = 0x00DA;
break; // U -> Ú
case 0x0075:
composed = 0x00FA;
break; // u -> ú
case 0x0059:
composed = 0x00DD;
break; // Y -> Ý
case 0x0079:
composed = 0x00FD;
break; // y -> ý
default:
break;
}
break;
case 0x0302: // circumflex
switch (prev) {
case 0x0041:
composed = 0x00C2;
break; // A -> Â
case 0x0061:
composed = 0x00E2;
break; // a -> â
case 0x0045:
composed = 0x00CA;
break; // E -> Ê
case 0x0065:
composed = 0x00EA;
break; // e -> ê
case 0x0049:
composed = 0x00CE;
break; // I -> Î
case 0x0069:
composed = 0x00EE;
break; // i -> î
case 0x004F:
composed = 0x00D4;
break; // O -> Ô
case 0x006F:
composed = 0x00F4;
break; // o -> ô
case 0x0055:
composed = 0x00DB;
break; // U -> Û
case 0x0075:
composed = 0x00FB;
break; // u -> û
default:
break;
}
break;
case 0x0303: // tilde
switch (prev) {
case 0x0041:
composed = 0x00C3;
break; // A -> Ã
case 0x0061:
composed = 0x00E3;
break; // a -> ã
case 0x004E:
composed = 0x00D1;
break; // N -> Ñ
case 0x006E:
composed = 0x00F1;
break; // n -> ñ
default:
break;
}
break;
case 0x0308: // diaeresis/umlaut
switch (prev) {
case 0x0041:
composed = 0x00C4;
break; // A -> Ä
case 0x0061:
composed = 0x00E4;
break; // a -> ä
case 0x0045:
composed = 0x00CB;
break; // E -> Ë
case 0x0065:
composed = 0x00EB;
break; // e -> ë
case 0x0049:
composed = 0x00CF;
break; // I -> Ï
case 0x0069:
composed = 0x00EF;
break; // i -> ï
case 0x004F:
composed = 0x00D6;
break; // O -> Ö
case 0x006F:
composed = 0x00F6;
break; // o -> ö
case 0x0055:
composed = 0x00DC;
break; // U -> Ü
case 0x0075:
composed = 0x00FC;
break; // u -> ü
case 0x0059:
composed = 0x0178;
break; // Y -> Ÿ
case 0x0079:
composed = 0x00FF;
break; // y -> ÿ
default:
break;
}
break;
case 0x0327: // cedilla
switch (prev) {
case 0x0043:
composed = 0x00C7;
break; // C -> Ç
case 0x0063:
composed = 0x00E7;
break; // c -> ç
default:
break;
}
break;
default:
break;
}
if (composed != 0) {
cps.back().value = composed;
continue; // skip pushing the combining mark itself
}
}
cps.push_back({cp, static_cast<size_t>(current - base)});
}
lib/Epub/Epub/hyphenation/Hyphenator.cpp
+49 -1
View File
@@ -1,8 +1,10 @@
#include "Hyphenator.h"
#include <algorithm>
#include <vector>
#include "HyphenationCommon.h"
#include "LanguageHyphenator.h"
#include "LanguageRegistry.h"
const LanguageHyphenator* Hyphenator::cachedHyphenator_ = nullptr;
@@ -32,10 +34,19 @@ size_t byteOffsetForIndex(const std::vector<CodepointInfo>& cps, const size_t in
}
// Builds a vector of break information from explicit hyphen markers in the given codepoints.
// Only hyphens that appear between two alphabetic characters are considered valid breaks.
//
// Example: "US-Satellitensystems" (cps: U, S, -, S, a, t, ...)
// -> finds '-' at index 2 with alphabetic neighbors 'S' and 'S'
// -> returns one BreakInfo at the byte offset of 'S' (the char after '-'),
// with requiresInsertedHyphen=false because '-' is already visible.
//
// Example: "Satel\u00ADliten" (soft-hyphen between 'l' and 'l')
// -> returns one BreakInfo with requiresInsertedHyphen=true (soft-hyphen
// is invisible and needs a visible '-' when the break is used).
std::vector<Hyphenator::BreakInfo> buildExplicitBreakInfos(const std::vector<CodepointInfo>& cps) {
std::vector<Hyphenator::BreakInfo> breaks;
// Scan every codepoint looking for explicit/soft hyphen markers that are surrounded by letters.
for (size_t i = 1; i + 1 < cps.size(); ++i) {
const uint32_t cp = cps[i].value;
if (!isExplicitHyphen(cp) || !isAlphabetic(cps[i - 1].value) || !isAlphabetic(cps[i + 1].value)) {
@@ -63,6 +74,43 @@ std::vector<Hyphenator::BreakInfo> Hyphenator::breakOffsets(const std::string& w
// Explicit hyphen markers (soft or hard) take precedence over language breaks.
auto explicitBreakInfos = buildExplicitBreakInfos(cps);
if (!explicitBreakInfos.empty()) {
// When a word contains explicit hyphens we also run Liang patterns on each alphabetic
// segment between them. Without this, "US-Satellitensystems" would only offer one split
// point (after "US-"), making it impossible to break mid-"Satellitensystems" even when
// "US-Satelliten-" would fit on the line.
//
// Example: "US-Satellitensystems"
// Segments: ["US", "Satellitensystems"]
// Explicit break: after "US-" -> @3 (no inserted hyphen)
// Pattern breaks on "Satellitensystems" -> @5 Sa|tel (+hyphen)
// @8 Satel|li (+hyphen)
// @10 Satelli|ten (+hyphen)
// @13 Satelliten|sys (+hyphen)
// @16 Satellitensys|tems (+hyphen)
// Result: 6 sorted break points; the line-breaker picks the widest prefix that fits.
if (hyphenator) {
size_t segStart = 0;
for (size_t i = 0; i <= cps.size(); ++i) {
const bool atEnd = (i == cps.size());
const bool atHyphen = !atEnd && isExplicitHyphen(cps[i].value);
if (atEnd || atHyphen) {
if (i > segStart) {
std::vector<CodepointInfo> segment(cps.begin() + segStart, cps.begin() + i);
auto segIndexes = hyphenator->breakIndexes(segment);
for (const size_t idx : segIndexes) {
const size_t cpIdx = segStart + idx;
if (cpIdx < cps.size()) {
explicitBreakInfos.push_back({cps[cpIdx].byteOffset, true});
}
}
}
segStart = i + 1;
}
}
// Merge explicit and pattern breaks into ascending byte-offset order.
std::sort(explicitBreakInfos.begin(), explicitBreakInfos.end(),
[](const BreakInfo& a, const BreakInfo& b) { return a.byteOffset < b.byteOffset; });
}
return explicitBreakInfos;
}
lib/Epub/Epub/hyphenation/Hyphenator.h
+17 -4
View File
@@ -9,11 +9,24 @@ class LanguageHyphenator;
class Hyphenator {
public:
struct BreakInfo {
size_t byteOffset;
bool requiresInsertedHyphen;
size_t byteOffset; // Byte position inside the UTF-8 word where a break may occur.
bool requiresInsertedHyphen; // true = a visible '-' must be rendered at the break (pattern/fallback breaks).
// false = the word already contains a hyphen at this position (explicit '-').
};
// Returns byte offsets where the word may be hyphenated. When includeFallback is true, all positions obeying the
// minimum prefix/suffix constraints are returned even if no language-specific rule matches.
// Returns byte offsets where the word may be hyphenated.
//
// Break sources (in priority order):
// 1. Explicit hyphens already present in the word (e.g. '-' or soft-hyphen U+00AD).
// When found, language patterns are additionally run on each alphabetic segment
// between hyphens so compound words can break within their parts.
// Example: "US-Satellitensystems" yields breaks after "US-" (no inserted hyphen)
// plus pattern breaks inside "Satellitensystems" (Sa|tel|li|ten|sys|tems).
// 2. Language-specific Liang patterns (e.g. German de_patterns).
// Example: "Quadratkilometer" -> Qua|drat|ki|lo|me|ter.
// 3. Fallback every-N-chars splitting (only when includeFallback is true AND no
// pattern breaks were found). Used as a last resort to prevent a single oversized
// word from overflowing the page width.
static std::vector<BreakInfo> breakOffsets(const std::string& word, bool includeFallback);
// Provide a publication-level language hint (e.g. "en", "en-US", "ru") used to select hyphenation rules.