The pseudobibeR package calculates the lexicogrammatical and functional features described by Biber (1985) and widely used for text-type, register, and genre classification tasks. The package extracts 67 different linguistic features from pre-parsed text data, enabling researchers to analyze linguistic variation across different text types and registers.
This package doesn't perform part-of-speech tagging itself. Instead, it leverages existing high-quality taggers from udpipe or spaCy (via spacyr) to extract and aggregate linguistic patterns. The accuracy of feature extraction depends on the quality of the underlying part-of-speech tagging and dependency parsing.
Note: Texts with irregular spellings, non-normative punctuation, or domain-specific language may produce less reliable outputs unless the taggers are specifically tuned for those purposes.
Install the stable version from CRAN:
install.packages("pseudobibeR")
Or install the development version from GitHub:
# install.packages("devtools")
# devtools::install_github("browndw/pseudobibeR")
install.packages("pseudobibeR", repos = c('https://browndw.r-universe.dev', 'https://cloud.r-project.org'))
The package provides the main function biber(), which takes either udpipe- or spacyr-tagged text and produces a data frame of linguistic features for each document.
library(spacyr)
library(pseudobibeR)
# Initialize spaCy (requires Python and spaCy installation)
spacy_initialize(model = "en_core_web_sm")
# Parse text and extract features
features <- biber(
spacy_parse(
c("doc_1" = "The task was done by Steve",
"doc_2" = "I think that you should reconsider your approach"),
dependency = TRUE,
tag = TRUE,
pos = TRUE
)
)
print(features)
library(udpipe)
library(pseudobibeR)
# Download and load udpipe model
ud_model <- udpipe_download_model(language = "english")
ud_model <- udpipe_load_model(ud_model$file_model)
# Parse text and extract features
text_data <- data.frame(
doc_id = c("doc_1", "doc_2"),
text = c("The task was done by Steve",
"I think that you should reconsider your approach")
)
parsed_data <- udpipe_annotate(ud_model,
x = text_data$text,
doc_id = text_data$doc_id,
tagger = "default",
parser = "default")
features <- biber(parsed_data)
print(features)
The package includes sample data to demonstrate functionality:
library(pseudobibeR)
# Extract features from sample spaCy-parsed data
spacy_features <- biber(spacy_samples)
head(spacy_features)
# Extract features from sample udpipe-parsed data
udpipe_features <- biber(udpipe_samples)
head(udpipe_features)
Choose one of the following parsing pipelines:
en_core_web_sm)See the respective package documentation for installation instructions.
The biber() function accepts the following arguments:
tokens: Pre-parsed text data from spacyr::spacy_parse() or udpipe::udpipe_annotate()
measure: Type-token ratio measure ("MATTR", "TTR", "CTTR", "MSTTR", or "none")normalize: Whether to normalize counts to per 1,000 tokens (TRUE/FALSE)# Example with custom parameters
features <- biber(parsed_data,
measure = "MATTR",
normalize = TRUE)
pseudobibeR uses testthat for comprehensive unit testing. To avoid distributing large language models with the package, tests use pre-saved parsing outputs.
tests/testthat/text-samples/samples.tsv: Sample sentences for testingtests/testthat/text-samples/parse-samples.R: Script to generate parsed samplestests/testthat/text-samples/*.rds: Saved parsing outputs for testsIf you modify samples.tsv, regenerate the parsed samples:
# Navigate to tests/testthat/text-samples/
source("parse-samples.R")
# Run all tests
testthat::test_package("pseudobibeR")
# Run specific test files
testthat::test_file("tests/testthat/test-features.R")
The package includes several built-in datasets:
dict: Dictionary patterns for feature detection (e.g., third-person pronouns, conjunctions)word_lists: Exact word lists for specific featuresspacy_samples: Sample spaCy-parsed text dataudpipe_samples: Sample udpipe-parsed text dataThese can be examined to understand feature definitions:
# View available dictionaries
names(dict)
# Examine specific word lists
word_lists$f_06_first_person_pronouns
# Load sample data
data(spacy_samples)
data(udpipe_samples)
When using pseudobibeR in your research, please cite:
The original Biber (1985) paper:
Biber, D. (1985). Investigating macroscopic textual variation through multifeature/multidimensional analyses. Linguistics, 23(2), 337-360. DOI: 10.1515/ling.1985.23.2.337
This package:
Brown, D. W. (2024). pseudobibeR: Aggregate Counts of Linguistic Features. R package version 1.1. https://github.com/browndw/pseudobibeR
The package extracts 67 linguistic features organized into 16 categories based on Biber (1985). Each feature is identified through a combination of part-of-speech patterns, dependency relations, and lexical matching.
| Category | Features | Description |
|---|---|---|
| A. Tense and aspect markers | f_01-f_03 | Past tense, perfect aspect, present tense |
| B. Place and time adverbials | f_04-f_05 | Spatial and temporal adverbials |
| C. Pronouns and pro-verbs | f_06-f_12 | Personal pronouns, demonstratives, indefinites, pro-verb do |
| D. Questions | f_13 | Direct wh-questions |
| E. Nominal forms | f_14-f_16 | Nominalizations, gerunds, other nouns |
| F. Passives | f_17-f_18 | Agentless and by-passives |
| G. Stative forms | f_19-f_20 | be as main verb, existential there |
| H. Subordination features | f_21-f_38 | Various subordinate clause types |
| I. Prepositional phrases, adjectives and adverbs | f_39-f_42 | Prepositional phrases, attributive/predicative adjectives, adverbs |
| J. Lexical specificity | f_43-f_44 | Type-token ratio, average word length |
| K. Lexical classes | f_45-f_51 | Conjuncts, downtoners, hedges, amplifiers, emphatics, etc. |
| L. Modals | f_52-f_54 | Possibility, necessity, and predictive modals |
| M. Specialized verb classes | f_55-f_58 | Public, private, suasive verbs, seem/appear |
| N. Reduced forms and dispreferred structures | f_59-f_63 | Contractions, deletions, split constructions |
| O. Co-ordination | f_64-f_65 | Phrasal and clausal coordination |
| P. Negation | f_66-f_67 | Synthetic and analytic negation |
| Feature | Description |
|---|---|
| - | A. Tense and aspect markers |
| f_01_past_tense | Past tense |
| f_02_perfect_aspect | Perfect aspect |
| f_03_present_tense | Present tense |
| - | B. Place and time adverbials |
| f_04_place_adverbials | Place adverbials (e.g., above, beside, outdoors) |
| f_05_time_adverbials | Time adverbials (e.g., early, instantly, soon) |
| - | C. Pronouns and pro-verbs |
| f_06_first_person_pronouns | First-person pronouns |
| f_07_second_person_pronouns | Second-person pronouns |
| f_08_third_person_pronouns | Third-person personal pronouns (excluding it) |
| f_09_pronoun_it | Pronoun it |
| f_10_demonstrative_pronoun | Demonstrative pronouns (that, this, these, those as pronouns) |
| f_11_indefinite_pronoun | Indefinite pronounes (e.g., anybody, nothing, someone) |
| f_12_proverb_do | Pro-verb do |
| - | D. Questions |
| f_13_wh_question | Direct wh-questions |
| - | E. Nominal forms |
| f_14_nominalization | Nominalizations (ending in -tion, -ment, -ness, -ity) |
| f_15_gerunds | Gerunds (participial forms functioning as nouns) |
| f_16_other_nouns | Total other nouns |
| - | F. Passives |
| f_17_agentless_passives | Agentless passives |
| f_18_by_passives | by-passives |
| - | G. Stative forms |
| f_19_be_main_verb | be as main verb |
| f_20_existential_there | Existential there |
| - | H. Subordination features |
| f_21_that_verb_comp | that verb complements (e.g., I said [that he went].) |
| f_22_that_adj_comp | that adjective complements (e.g., I'm glad [that you like it].) |
| f_23_wh_clause | wh-clauses (e.g., I believed [what he told me].) |
| f_24_infinitives | Infinitives |
| f_25_present_participle | Present participial adverbial clauses (e.g., [Stuffing his mouth with cookies], Joe ran out the door.) |
| f_26_past_participle | Past participial adverbial clauses (e.g., [Built in a single week], the house would stand for fifty years.) |
| f_27_past_participle_whiz | Past participial postnominal (reduced relative) clauses (e.g., the solution [produced by this process]) |
| f_28_present_participle_whiz | Present participial postnominal (reduced relative) clauses (e.g., the event [causing this decline[) |
| f_29_that_subj | that relative clauses on subject position (e.g., the dog [that bit me]) |
| f_30_that_obj | that relative clauses on object position (e.g., the dog [that I saw]) |
| f_31_wh_subj | wh- relatives on subject position (e.g., the man [who likes popcorn]) |
| f_32_wh_obj | wh- relatives on object position (e.g., the man [who Sally likes]) |
| f_33_pied_piping | Pied-piping relative clauses (e.g., the manner [in which he was told]) |
| f_34_sentence_relatives | Sentence relatives (e.g., Bob likes fried mangoes, [which is the most disgusting thing I've ever heard of].) |
| f_35_because | Causative adverbial subordinator (because) |
| f_36_though | Concessive adverbial subordinators (although, though) |
| f_37_if | Conditional adverbial subordinators (if, unless) |
| f_38_other_adv_sub | Other adverbial subordinators (e.g., since, while, whereas) |
| - | I. Prepositional phrases, adjectives and adverbs |
| f_39_prepositions | Total prepositional phrases |
| f_40_adj_attr | Attributive adjectives (e.g., the [big] horse) |
| f_41_adj_pred | Predicative adjectives (e.g., The horse is [big].) |
| f_42_adverbs | Total adverbs |
| - | J. Lexical specificity |
| f_43_type_token | Type-token ratio (including punctuation) |
| f_44_mean_word_length | Average word length (across tokens, excluding punctuation) |
| - | K. Lexical classes |
| f_45_conjuncts | Conjuncts (e.g., consequently, furthermore, however) |
| f_46_downtoners | Downtoners (e.g., barely, nearly, slightly) |
| f_47_hedges | Hedges (e.g., at about, something like, almost) |
| f_48_amplifiers | Amplifiers (e.g., absolutely, extremely, perfectly) |
| f_49_emphatics | Emphatics (e.g., a lot, for sure, really) |
| f_50_discourse_particles | Discourse particles (e.g., sentence-initial well, now, anyway) |
| f_51_demonstratives | Demonstratives |
| - | L. Modals |
| f_52_modal_possibility | Possibility modals (can, may, might, could) |
| f_53_modal_necessity | Necessity modals (ought, should, must) |
| f_54_modal_predictive | Predictive modals (will, would, shall) |
| - | M. Specialized verb classes |
| f_55_verb_public | Public verbs (e.g., assert, declare, mention) |
| f_56_verb_private | Private verbs (e.g., assume, believe, doubt, know) |
| f_57_verb_suasive | Suasive verbs (e.g., command, insist, propose) |
| f_58_verb_seem | seem and appear |
| - | N. Reduced forms and dispreferred structures |
| f_59_contractions | Contractions |
| f_60_that_deletion | Subordinator that deletion (e.g., I think [he went].) |
| f_61_stranded_preposition | Stranded prepositions (e.g., the candidate that I was thinking [of]) |
| f_62_split_infinitve | Split infinitives (e.g., He wants [to convincingly prove] that …) |
| f_63_split_auxiliary | Split auxiliaries (e.g., They [were apparently shown] to …) |
| - | O. Co-ordination |
| f_64_phrasal_coordination | Phrasal co-ordination (N and N; Adj and Adj; V and V; Adv and Adv) |
| f_65_clausal_coordination | Independent clause co-ordination (clause-initial and) |
| - | P. Negation |
| f_66_neg_synthetic | Synthetic negation (e.g., No answer is good enough for Jones.) |
| f_67_neg_analytic | Analytic negation (e.g., That isn't good enough.) |
Contributions are welcome! Please feel free to submit issues or pull requests on GitHub.
When reporting bugs or requesting features, please include:
This package is licensed under the MIT License. See the LICENSE file for details.