Analyze directional preferences (left vs right) for collocations.

The original Daudaravičius & Marcinkevičienė formula. This is the main contribution of their paper.

PMI-weighted gravity (alternative formulation). G = f(w1,w2) × PMI(w1,w2)

Simplified gravity formula often used in implementations. G = log₂((f(w1,w2)² × span) / (f(w1) × f(w2)))

aggregate_contingency_tables(tables::Vector{ContingencyTable}, minfreq::Int) -> DataFrame

Aggregate multiple contingency tables into a single table.

analyze_corpus(corpus::Corpus, node::AbstractString, metric::Type{<:AssociationMetric};
              windowsize::Int, minfreq::Int=5) -> DataFrame

Analyze a single node word across the entire corpus using corpus's normalization. Returns DataFrame with Node, Collocate, Score, Frequency, and DocFrequency columns.

analyze_nodes(corpus::Corpus, nodes::Vector{String}, metrics::Vector{DataType};
             windowsize::Int, minfreq::Int=5, top_n::Int=100,
             parallel::Bool=false) -> MultiNodeAnalysis

Analyze multiple nodes with consistent normalization. Each result DataFrame now includes the Node column and metadata.

analyze_temporal(corpus::Corpus,
                        nodes::Vector{String},
                        time_field::Symbol,
                        metric::Type{<:AssociationMetric};
                        time_bins::Int=10,
                        windowsize::Int=5,
                        minfreq::Int=5) -> TemporalCorpusAnalysis

Analyze how word associations change over time.

assoc_score(metrics::AbstractVector{<:Type{<:AssociationMetric}},
          inputstring::AbstractString,
          node::AbstractString;
          windowsize::Int,
          minfreq::Int=5;
          scores_only::Bool=false,
          norm_config::TextNorm=TextNorm(),
          tokens::Union{Nothing,Vector{String}}=nothing,
          kwargs...)

Convenience overload to compute multiple metrics directly from raw text.

assoc_score(metrics::AbstractVector{<:Type{<:AssociationMetric}},
          x::AssociationDataFormat;
          scores_only::Bool=false,
          tokens::Union{Nothing,Vector{String}}=nothing,
          kwargs...)

Evaluate multiple metrics on CT or CCT.

• Returns a DataFrame with one column per metric by default.
• If scores_only=true, returns Dict{String,Vector{Float64}}.

assoc_score(metrics::AbstractVector{<:Type{<:AssociationMetric}}, 
            corpus::Corpus, nodes::Vector{String}; 
            windowsize::Int=5, minfreq::Int=5, top_n::Int=100, kwargs...)

Evaluate multiple metrics on multiple nodes in a corpus. Returns a Dict{String,DataFrame} with combined metric results for each node.

assoc_score(metricType::Type{<:AssociationMetric},
          inputstring::AbstractString,
          node::AbstractString;
          windowsize::Int,
          minfreq::Int=5;
          scores_only::Bool=false,
          norm_config::TextNorm=TextNorm(),
          tokens::Union{Nothing,Vector{String}}=nothing,
          kwargs...)

Convenience overload to compute a metric directly from a raw string.

assoc_score(metricType::Type{<:AssociationMetric}, x::AssociationDataFormat;
          scores_only::Bool=false,
          tokens::Union{Nothing,Vector{String}}=nothing,
          kwargs...)

Evaluate a metric on any association data format (CT or CCT).

• If the metric requires tokens (e.g., LexicalGravity), pass tokens=... or implement assoc_tokens(::YourType) to supply them automatically.
• Returns a DataFrame by default: [:Node, :Collocate, :Frequency, :<MetricName>].
• If scores_only=true, returns only the scores Vector.

assoc_score(metricType::Type{<:AssociationMetric}, corpus::Corpus, node::AbstractString;
            windowsize::Int=5, minfreq::Int=5, kwargs...)

Evaluate a metric on a corpus - convenience method that delegates to analyze_corpus.

assoc_score(metricType::Type{<:AssociationMetric}, corpus::Corpus, 
            nodes::Vector{String}; windowsize::Int=5, minfreq::Int=5, 
            top_n::Int=100, kwargs...)

Evaluate a metric on multiple nodes in a corpus. Returns a Dict{String,DataFrame} with results for each node.

assoc_score(metric::Type{<:AssociationMetric}, corpus::Corpus;
            nodes::Vector{String}, windowsize::Int=5, minfreq::Int=5,
            top_n::Int=100, kwargs...)

Alternative syntax with nodes as keyword argument.

available_metrics() -> Vector{DataType}

Returns a list of all supported association metrics.

batch_process_corpus(corpus::Corpus,
                    node_file::AbstractString,
                    output_dir::AbstractString;
                    metrics::Vector{DataType}=[PMI, LogDice],
                    windowsize::Int,
                    minfreq::Int=5,
                    batch_size::Int=100)

Process a large list of node words in batches. Results include Node column.

build_document_term_matrix(documents, vocabulary) -> SparseMatrixCSC

Build a document-term matrix from documents.

build_vocab(input::Union{StringDocument,Vector{String}}) -> OrderedDict

Create vocabulary dictionary from text input.

cached_data(z::LazyProcess{T,R}) -> R

Extract data from a LazyProcess, computing it if necessary.

calculate_effect_sizes(results, metric) -> DataFrame

Calculate effect sizes for differences between subcorpora.

colloc_graph(corpus::Corpus,
                        seed_words::Vector{String};
                        metric::Type{<:AssociationMetric}=PMI,
                        depth::Int=2,
                        min_score::Float64=3.0,
                        max_neighbors::Int=20) -> CollocationNetwork

Build a collocation network starting from seed words.

compare_subcorpora(corpus::Corpus,
                  split_field::Symbol,
                  node::String,
                  metric::Type{<:AssociationMetric};
                  windowsize::Int=5,
                  minfreq::Int=5) -> SubcorpusComparison

Compare word associations across different subcorpora.

compute_association_trends(results_by_period, nodes, metric) -> DataFrame

Compute trend statistics for associations over time.

cont_table(input_doc::StringDocument, target_word::AbstractString,
          windowsize::Int=5, minfreq::Int=3) -> DataFrame

Compute the contingency table for a target word in a document. Note: target_word should already be normalized before calling this function.

corpus_stats(corpus::Corpus; 
                 include_token_distribution::Bool=true) -> Dict

Get comprehensive statistics about the corpus.

count_substrings(text::String, substring::String) -> Int

Count occurrences of a substring in text.

count_substrings(text::String, substrings::Vector{String}) -> Dict{String,Int}

Count occurrences of multiple substrings in text.

count_word_frequency(doc::StringDocument, word::String) -> Int

Count the frequency of a word in the document.

coverage_summary(stats::Dict)

Pretty print the vocabulary coverage statistics.

document(input::LazyInput) -> StringDocument

Extract the document from a LazyInput wrapper.

eval_lexicalgravity(data::AssociationDataFormat; 
                   tokens::Vector{String},
                   formula::Symbol=:original)

Compute Lexical Gravity measure based on Daudaravičius & Marcinkevičienė (2004).

Arguments

• data: AssociationDataFormat with co-occurrence data
• tokens: Required tokenized text (provided by assoc_score when called through API)
• formula: Which formula to use:
  • :original - The main formula from the paper: G→(w1,w2) = log(f→×n+/f1) + log(f←×n-/f2)
  • :simplified - Simplified version: G = log₂((f²×span)/(f1×f2))
  • :pmi_weighted - PMI-style weighting: G = f(w1,w2) × log((f×N)/(f1×f2))

Original Formula from Paper

G→(w1,w2) = log(f→(w1,w2)/f(w1) × n+(w1)) + log(f←(w1,w2)/f(w2) × n-(w2))

Where:

• f→(w1,w2) = frequency of w2 following w1 within window
• f←(w1,w2) = frequency of w1 preceding w2 within window
• n+(w1) = number of different word types that follow w1
• n-(w2) = number of different word types that precede w2
• f(w1), f(w2) = total frequencies of words

Note

This function expects tokens to be provided. When called through assoc_score(), tokens are automatically fetched based on the NeedsTokens trait.

extract_rake_keywords(corpus, num_keywords) -> DataFrame

Extract keywords using RAKE algorithm (placeholder).

extract_textrank_keywords(corpus, num_keywords) -> DataFrame

Extract keywords using TextRank algorithm (placeholder).

extract_tfidf_keywords(corpus, num_keywords, min_doc_freq, max_doc_freq_ratio) -> DataFrame

Extract keywords using TF-IDF scoring.

find_following_words(doc::StringDocument, word::String, window::Int) -> Set{String}

Find unique words that appear within window words after each occurrence of word in the document.

find_prior_words(doc::StringDocument, word::String, window::Int) -> Set{String}

Find unique words that appear within window words before each occurrence of word in the document.

gephi_graph(network::CollocationNetwork,
                       nodes_file::String,
                       edges_file::String)

Export network for visualization in Gephi or similar tools.

keyterms(corpus::Corpus;
                method::Symbol=:tfidf,
                num_keywords::Int=50,
                min_doc_freq::Int=3,
                max_doc_freq_ratio::Float64=0.5) -> DataFrame

Extract keywords from corpus using various methods.

kwic(corpus::Corpus,
                    node::String;
                    context_size::Int=50,
                    max_lines::Int=1000) -> Concordance

Generate KWIC concordance for a node word.

lexical_gravity_analysis(data::AssociationDataFormat; 
                        tokens::Union{Nothing,Vector{String}}=nothing)

Comprehensive analysis using all gravity formulas for comparison. Returns results from all three formulas plus directional analysis.

Note: If tokens are not provided, will attempt to fetch them using assoc_tokens.

Safe logarithm that handles zero and negative values.

normalize_node(node::AbstractString, config::TextNorm) -> String

Normalize a node word according to the given TextNorm configuration. This is the single source of truth for node normalization.

perform_statistical_tests(results, metric) -> DataFrame

Perform statistical tests between subcorpora.

prep_string(input_path::AbstractString, config::TextNorm) -> StringDocument

Prepare and preprocess text from various sources into a StringDocument.

Arguments

• input_path: File path, directory path, or raw text string.

Preprocessing options

Uses TextNorm configuration for all preprocessing options.

Returns

A preprocessed StringDocument object suitable for downstream corpus analysis.

read_corpus(path::AbstractString; kwargs...) -> Corpus

Load a corpus from various sources with consistent normalization.

read_corpus_df(df::DataFrame; kwargs...) -> Corpus

Load corpus directly from a DataFrame with consistent normalization. Metadata columns are stored at the corpus level with document indices.

stream_corpus_analysis(file_pattern::AbstractString,
                      node::AbstractString,
                      metric::Type{<:AssociationMetric};
                      windowsize::Int,
                      chunk_size::Int=1000)

Stream-process large corpora without loading everything into memory.

strip_diacritics(s::AbstractString; target_form::Symbol = :NFC) -> String

Remove all combining diacritics (e.g., Greek tonos, dialytika) using Unicode normalization.

Internally:

• canonically decomposes as needed,
• strips combining marks (Mn),
• and normalizes to target_form (default :NFC).

If you don’t care about the final form, leave target_form at the default.

Example: julia> strip_diacritics("ένα το χελιδόϊι") "ενα το χελιδοιι"

token_distribution(text::AbstractString) -> DataFrame

Analyze the distribution of tokens in a string and return a DataFrame with info about absolute and relative token frequencies.

token_distribution(corpus::Corpus) -> DataFrame

Analyze the distribution of tokens in the corpus.

Convert input to string vector for vocabulary creation.

vocab_coverage(corpus::Corpus; 
                         percentiles=0.01:0.01:1.0) -> DataFrame

Calculate vocabulary coverage curve showing how many words are needed to cover various percentages of the corpus. Uses the corpus vocabulary for consistent calculations.

write_results(analysis::MultiNodeAnalysis, path::AbstractString; format::Symbol=:csv)

Export analysis results to file. Results now include Node column.