textpress 1.1.1
Documentation
- Six vignettes added covering the full pipeline: web data, Wikipedia
data, regex search, dictionary search, semantic search (RAG), and basic
NLP processing.
- Basic NLP vignette walks through
nlp_split_sentences(),
nlp_tokenize_text() (word and Biber methods), and
nlp_cast_tokens() stepwise and as a single pipe.
- README revamped: tighter intro, API map, RAG/agent positioning,
vignette links.
Changes
util_fetch_embeddings() re-added for embedding
generation via Hugging Face inference endpoints (reversed 1.1.0 removal;
now calls the HF inference API rather than loading models locally).nlp_cast_tokens() documented and surfaced – flattens
the token list from nlp_tokenize_text() into a long-format
data frame with optional character spans.- Suggests trimmed:
ellmer and unused packages
removed.
textpress 1.1.0
API and naming
- Package is now organized around a four-stage
pipeline: Fetch → Read → Process → Search. All functions use a
consistent verb_noun pattern.
- Acquire:
fetch_urls() (from web
search), fetch_wiki_urls(), fetch_wiki_refs()
— return URLs or metadata, not full text.
- Ingest:
read_urls() — read content
from URLs into R (replaces web_scrape_urls).
- Process:
nlp_split_*,
nlp_tokenize_text(), nlp_index_tokens() (and
nlp_roll_chunks() for rolling windows).
- Search: Four retrieval options —
search_regex() (regex/KWIC), search_index()
(BM25), search_vector() (cosine over your own embeddings),
search_dict() (dictionary match; replaces
ner_extract_entities).
- Common parameters standardized:
corpus (replaces
tif), by (replaces
text_hierarchy).
Removed
- In-package embedding generation (e.g. Hugging Face API). Use your
own embedding pipeline and pass your embedding matrix as the argument to
.
- Legacy names:
web_search, wiki_search,
wiki_find_references, web_scrape_urls,
ner_extract_entities, sem_nearest_neighbors /
sem_search_corpus (replaced by search_vector
and search_regex).
Docs
- README revamped around the API map and a single “golden path”
workflow.
- DESCRIPTION and package help updated for the four-stage pipeline;
version set to 1.1.0.
textpress 1.0.0
- Initial release: URL fetching, URL content reading, NLP processing
(split, tokenize, index), and corpus/search utilities.