feat: dict based Hyphenation (#305)

## Summary

* Adds (optional) Hyphenation for English, French, German, Russian
languages

## Additional Context

* Included hyphenation dictionaries add approximately 280kb to the flash
usage (German alone takes 200kb)
* Trie encoded dictionaries are adopted from hypher project
(https://github.com/typst/hypher)
* Soft hyphens (and other explicit hyphens) take precedence over
dict-based hyphenation. Overall, the hyphenation rules are quite
aggressive, as I believe it makes more sense on our smaller screen.

---------

Co-authored-by: Dave Allie <dave@daveallie.com>
This commit is contained in:
Arthur Tazhitdinov
2026-01-19 12:56:26 +00:00
committed by GitHub
co-authored by Dave Allie
parent 5fef99c641
commit 8824c87490
40 changed files with 36465 additions and 52 deletions
@@ -0,0 +1,26 @@
#pragma once
#include <cstddef>
#include <string>
#include "LanguageHyphenator.h"
struct LanguageEntry {
const char* cliName;
const char* primaryTag;
const LanguageHyphenator* hyphenator;
};
struct LanguageEntryView {
const LanguageEntry* data;
size_t size;
const LanguageEntry* begin() const { return data; }
const LanguageEntry* end() const { return data + size; }
};
// Returns the Liang-backed hyphenator for a given primary language tag (e.g., "en", "fr").
const LanguageHyphenator* getLanguageHyphenatorForPrimaryTag(const std::string& primaryTag);
// Exposes the list of supported languages primarily for tooling/tests.
LanguageEntryView getLanguageEntries();