Text Dispersion & Lexical Richness TTR Analyzer
Calculate Type-Token Ratio (TTR), Guiraud Root TTR, Carroll's Index, Herdan's C, Hapax Legomena, and positional dispersion tracking across text segments.
Corpus & Tokenization Engine
Divides corpus chronologically into equal token segments to analyze vocabulary pacing and repetition drift.
Diversity Analytics & Index Scorecard
Very High Diversity93.9%
V / N = 0.9396.57
V / √N46
Vocabulary49
Running Words4.647
Formula: V / √(2N)0.984
Formula: log V / log N43 (93%)
Dis Legomena: 3| Word Type | Frequency | Relative Density |
|---|---|---|
| lexical | 2 | 4.08% |
| syntactic | 2 | 4.08% |
| and | 2 | 4.08% |
| empirical | 1 | 2.04% |
| observation | 1 | 2.04% |
| of | 1 | 2.04% |
| diversity | 1 | 2.04% |
| demonstrates | 1 | 2.04% |
| significant | 1 | 2.04% |
| variance | 1 | 2.04% |
| across | 1 | 2.04% |
| diverse | 1 | 2.04% |
| discourse | 1 | 2.04% |
| communities | 1 | 2.04% |
| sociolinguistic | 1 | 2.04% |
Linguistic Insight: For manuscripts exceeding 500 words, compare Guiraud's Root TTR (V / √N) and Herdan's Index rather than raw TTR. Raw TTR naturally decreases with length due to unavoidable repetition of grammatical particles and determiners.
The Mathematical Mechanics of Lexical Diversity & Type-Token Ratio
Lexical richness describes the variety, depth, and non-redundancy of vocabulary deployed in a text corpus. While an initial pass with a readability score calculator evaluates cognitive ease and grade levels, calculating the Type-Token Ratio (TTR) establishes an objective baseline for stylistic maturity, genre variation, authorship attribution, and linguistic entropy.
Types vs. Tokens
A “token” represents any single instance of a word occurring in running text, whereas a “type” represents a unique word form. The sentence “The dog chased the cat” contains 5 tokens but only 4 types because the determiner “the” repeats twice.
Non-Linear Decay
Because languages operate on closed grammatical classes (prepositions, conjunctions, pronouns), new vocabulary introduces at sub-linear rates as document length increases. As a consequence, uncorrected TTR slopes downward naturally over longer texts.
Singleton Frequencies
Hapax legomena (words occurring exactly once) serve as the primary engine of creative differentiation. A high ratio of singletons indicates deliberate word choice, descriptive nuance, and technical domain specificity.
Corpus Formulas & Computational Definitions
| Metric Name | Algebraic Formula | Core Stylometric Purpose |
|---|---|---|
| Standard TTR | TTR = V / N | Baseline ratio of unique types (V) to total tokens (N). Optimal for short passages (50–300 words). |
| Guiraud's Root TTR | R = V / √N | Compensates for sample size by scaling against the square root of total tokens. Standard in stylometric benchmarks. |
| Carroll's CTTR | CTTR = V / √(2N) | Reduces sample-size distortion across long prose by normalizing against twice the corpus token volume. |
| Herdan's C Index | C = log(V) / log(N) | Bilogarithmic formulation assessing relative vocabulary growth rate across book-length manuscripts. |
| Hapax Ratio | HR = V₁ / V | Measures singleton rarity; calculates the percentage of total vocabulary consisting of words used only once. |
Comparative Benchmark Matrix: Lexical Richness by Genre & Medium
Lexical density and diversity vary drastically across communication mediums. While tools like a word frequency and stopword counter isolate prominent thematic terms and raw keyword volume, stylistic requirements differ: technical documentation prioritizes exact terminology, whereas creative narratives depend upon broad lexical variation:
| Writing Genre | Expected TTR (300w) | Guiraud Index | Dominant Structural Characteristics | Analytical Pitfalls |
|---|---|---|---|---|
| Academic Journals | 0.65 – 0.80 | 8.5 – 12.0 | High nominal density, polysyllabic Latinate roots, low pronoun counts | Overuse of passive voice and standard jargon clusters |
| Literary Fiction | 0.55 – 0.72 | 7.5 – 10.5 | Sensory adjectives, evocative metaphors, varied dialogue verbs | Dialogue passages naturally depress overall TTR |
| Product Copywriting | 0.40 – 0.55 | 5.0 – 7.5 | Purposeful message reinforcement, recurring brand keywords, action verbs | Deliberate repetition can be mistaken for vocabulary deficiency |
| SEO & Web Articles | 0.50 – 0.65 | 6.8 – 9.2 | Semantic entity expansion, conversational readability, structured headings | Keyword stuffing creates artificial localized repetition spikes |
Practical Stylistic Enhancement: 5 Steps to Balance Vocabulary Richness
Achieving high lexical diversity does not mean replacing everyday expressions with esoteric thesaurus substitutes. Follow these actionable editing principles to balance clarity, engagement, and vocabulary breadth:
Recommended Editing Practices
- • Analyze Sequential Slices: Check your segment dispersion graph to identify sudden dips in vocabulary variety, which often signal repetitive exposition or tired phrasing.
- • Replace Vague Dummy Verbs:Swap generic verbs (“make”, “get”, “do”, “have”) with active, descriptive counterparts (“formulate”, “procure”, “execute”).
- • Toggle Stop Word Filtering: Turn on stop word exclusion to analyze your core lexical nouns and verbs independently from grammatical glue words.
- • Vary Sentence Openers:Repeated sentence starters (“This shows...”, “It is...”) bloat pronoun frequencies and depress Guiraud ratings.
- • Leverage Semantic Co-occurrence: In digital content, use related domain concepts and LSI synonyms rather than repeating the exact target query.
Vocabulary Pitfalls to Avoid
- • The Thesaurus Trap: Inserting obscure synonyms can increase raw TTR while destroying readability and natural rhythmic cadence.
- • Ignoring Length Distortion:Never compare the raw TTR of a 2,000-word article directly against a 150-word abstract without using Guiraud's or Herdan's normalized indices.
- • Dialogue Stripping Blindspots: Natural human speech is naturally repetitive. Novel chapters with heavy character dialogue will register lower diversity without indicating poor writing.
- • Brand Name Dilution: In commercial writing, maintaining consistent product names is crucial for consumer clarity, even if it introduces localized repetition.
Frequently Asked Questions (FAQ)
What is Type-Token Ratio (TTR) in quantitative linguistics?
Type-Token Ratio (TTR) is a computational metric of lexical diversity defined as the ratio of unique word types (V) to total running word tokens (N). A higher ratio indicates a rich, varied vocabulary, while a lower ratio indicates recurrent vocabulary and thematic repetition.
Why does standard TTR decline as text length increases?
Standard TTR suffers from sample-size dependency. As a text grows longer, function words (such as articles, prepositions, and pronouns) repeat inevitably, causing the total token count (N) to outpace new unique vocabulary introductions (V). This is why linguists apply sample-size corrected formulations like Guiraud's Root TTR and Herdan's bilogarithmic C.
What is the difference between Guiraud's Root TTR and Carroll's Corrected TTR?
Guiraud's Root TTR divides unique types by the square root of total tokens (V / √N), diminishing the distorting effect of text length. Carroll's Corrected TTR divides types by the square root of twice the token count (V / √(2N)), offering an alternative normalization factor commonly utilized in comparative stylometry.
What are Hapax Legomena and Dis Legomena?
Hapax Legomena are words that occur exactly once within a corpus. Dis Legomena are words that appear exactly twice. In quantitative corpus analysis, a higher proportion of Hapax Legomena relative to vocabulary size signals exceptional expressive nuance and specialized terminology.
How does segment dispersion help evaluate textual pacing?
Textual dispersion breaks a manuscript or corpus into sequential proportional segments. By calculating lexical richness across consecutive sections, writers and researchers can spot localized vocabulary exhaustion, dialogue shifts, or topical transitions where repetition spikes.
Does filtering stop words provide a more accurate measure of vocabulary?
Filtering grammatical stop words focuses the analysis exclusively on lexical content words (nouns, descriptive adjectives, verbs, and adverbs). This isolates semantic richness from structural syntactic frequency, delivering clearer insights into technical and thematic sophistication.
Related & Complementary Utilities
Explore more privacy-first client-side web tools.
Line Alphabetizer & Sort Suite
Sort, deduplicate, trim, and format line-delimited text instantly.
Invisible Whitespace & Zero-Width Space Cleaner
Detect, visualize, and strip hidden Unicode characters, zero-width spaces, BOM markers, and BiDi overrides in real time.
Text Syllable Stress & Iambic Pentameter Metric Poetry Auditor
Analyze poetic scansion, count syllables, calculate iambic foot ratios, and verify blank verse or Shakespearean sonnet meter.
Regular Expression Text Pattern Matcher & Group Extractor
Test regular expressions in real-time, extract matching text groups, perform search and replace operations, and debug RegEx patterns instantly.