Jeremy Klein
2823a4a2cd
Based on the learnings from https://github.com/crosspoint-reader/crosspoint-reader/pull/2074 , I wanted to bring the same buffer savings to the rest of our HTTP Client stack. That being said, HttpDownloader (fonts/OPDS) used the Arduino HTTPClient. HttpDownloader was the last consumer of the Arduino HTTPClient + NetworkClientSecure stack. OtaUpdater already runs on esp_http_client, so this drops the parallel HTTP/TLS implementation. It also fixes a class of OPDS/font download failures: HTTPClient's setTimeout is uint16 and truncates, and its short per-read deadline killed slow or chunked responses (the -11 / incomplete-data errors). What changed: - Rewrote fetchUrl/downloadToFile around esp_http_client with a streaming open() -> fetch_headers() -> read() loop, manual redirect following, and is_complete_data_received() as the completeness gate. Body bytes go straight to the sink (OPDS parser stream, std::string, or file), so nothing buffers the payload. - HTTPS is now verified against the CA bundle instead of NetworkClientSecure::setInsecure(). esp-tls is built with CONFIG_ESP_TLS_INSECURE off, so an unverified handshake can't be set up anyway; the model is public servers over verified https and local servers over plain http (transport is chosen from the URL scheme). ** Self-signed https servers are no longer supported, by design. ** - timeout_ms is 60s; esp_http_client's timeout is uint32, so unlike HTTPClient it doesn't silently truncate. - HTTP buffers are 4096 (rx) / 1024 (tx). 4096 holds real OPDS server headers; the GitHub release CDN sends more and logs a non-fatal truncation warning, but the headers we read (Location, Content-Length) come first and survive. - Removed the now-unused UrlUtils::isHttpsUrl and a stale HTTPClient comment in FontDownloadActivity. Validated on device: OPDS browse and a 3.4 MB book download over verified https, GitHub font downloads (crc-checked), redirect handling matching curl, and slow/erroring servers surfaced correctly. Did you use AI tools to help write this code? partial
51 lines
1.5 KiB
C++
51 lines
1.5 KiB
C++
#include "UrlUtils.h"
|
|
|
|
namespace UrlUtils {
|
|
|
|
std::string ensureProtocol(const std::string& url) {
|
|
if (url.find("://") == std::string::npos) {
|
|
return "http://" + url;
|
|
}
|
|
return url;
|
|
}
|
|
|
|
std::string extractHost(const std::string& url) {
|
|
const size_t protocolEnd = url.find("://");
|
|
if (protocolEnd == std::string::npos) {
|
|
// No protocol, find first slash
|
|
const size_t firstSlash = url.find('/');
|
|
return firstSlash == std::string::npos ? url : url.substr(0, firstSlash);
|
|
}
|
|
// Find the first slash after the protocol
|
|
const size_t hostStart = protocolEnd + 3;
|
|
const size_t pathStart = url.find('/', hostStart);
|
|
return pathStart == std::string::npos ? url : url.substr(0, pathStart);
|
|
}
|
|
|
|
std::string buildUrl(const std::string& serverUrl, const std::string& path) {
|
|
// If path is already an absolute URL (has protocol), use it directly
|
|
if (path.find("://") != std::string::npos) {
|
|
return path;
|
|
}
|
|
const std::string urlWithProtocol = ensureProtocol(serverUrl);
|
|
if (path.empty()) {
|
|
return urlWithProtocol;
|
|
}
|
|
if (path[0] == '/') {
|
|
// Absolute path - use just the host
|
|
return extractHost(urlWithProtocol) + path;
|
|
}
|
|
// Relative path - strip query string from base before appending
|
|
std::string base = urlWithProtocol;
|
|
const size_t queryPos = base.find('?');
|
|
if (queryPos != std::string::npos) {
|
|
base.resize(queryPos);
|
|
}
|
|
if (base.back() == '/') {
|
|
return base + path;
|
|
}
|
|
return base + "/" + path;
|
|
}
|
|
|
|
} // namespace UrlUtils
|