MediaFranca:A Practice-Oriented Investigation into a Generative Pictographic System for Cognitive Accessibility

Candidate

Herbert Spencer González
PhD Student in Design

Supervisors

Dr. Marcos Steagall

Dr. Ivana Nakarada-Kordic

Advisor

Dr. Welby Ings

Travesía 2018, Corredor Litoral.
First Year Design Studio + Third year Light and Color Design Studio

Research Centre for Accessibility and Inclusion + friends
Pontifical Catholic University of Valparaíso

research question:

How can a generative pictographic system be designed to support communication for people with complex communication needs?

Why this research matters?

between 6% and 10% have complex communication needs

Universal Accessibility Pictogram + a speaking bubble with a check sign

Augmentative and Alternative Communication (AAC)

AAC is an Interdisciplinary field and a community of practice that combines technologies, symbols, and interaction strategies to support and extend communication by adding new channels, enabling individuals with complex communication needs to participate meaningfully in social and communicative life.

A persistent gap

Speech & Language Augmentation	Predictive Communication Systems	Embodied / Sensor-Based Interfaces	Pictographic & Symbolic Representation
voice recognition text-to-speech synthesis speech-to-text captioning AI speech repair for dysarthria	predictive text next pictogram prediction context-aware phrase suggestion adaptive keyboards	swipe keyboards eye tracking gesture control facial expression detection EEG-based intent sensing	static pictogram sets no dynamic generative pictographic systems

Part 2
Contextual Review

The Tower of Babel, oil on panel by Pieter Bruegel the Elder, 1563.

Begriffsschrift, diagrammatic notation from Concept Script, Gottlob Frege, 1879.

Universal-local paradox

Illustration from “Language, Thought, and Reality: Selected Writings of Benjamin Lee Whorf”, 1956

The "Maluma-takete" effect (later renamed “Bouba-kiki”), Köhler, 1929.

65 semantic primes

Substantives

I
YOU
SOMEONE / PERSON
SOMETHING / THING
BODY
PEOPLE

Determiners & Quantifiers

THIS
THE SAME
OTHER / ELSE
ONE
TWO
MANY / MUCH
SOME / A FEW
ALL
THERE IS / EXISTS
HAVE (PARTS)

Actions & Events

DO
HAPPEN
MOVE
TOUCH

Mental Predicates

THINK
KNOW
WANT
FEEL
SEE
HEAR

Speech

SAY
WORD
TRUE

Time & Place

WHEN / TIME
NOW
BEFORE
AFTER
A LONG TIME
A SHORT TIME
WHERE / PLACE
HERE
ABOVE
BELOW
FAR
NEAR
SIDE
INSIDE

Logical Concepts

NOT / NO
MAYBE
CAN
BECAUSE
IF

Evaluators & Descriptors

GOOD
BAD
BIG
SMALL
VERY
LIKE / AS

Natural Semantic Metalanguage: Set of 65 semantic primes. London: Oxford University Press.
Wierzbicka, A. (1996, revised in 2017).

Conceptual blending theory

Part 3
First Concepts

Text as Image: The Aesthetics of Accessibility

Part 3
From Theory to Practice

The Oath of the Horatii, oil on canvas by Jacques-Louis David, 1784.

Blissymbolics chart, selection of ideographic symbols from Charles K. Bliss’s system of semantography, first published in the 1949.

PECS communication book, example of the Picture Exchange Communication System widely used in Chilean special education contexts. The system is based on the physical exchange of pictograms to initiate communicative acts and develop functional communication in individuals with autism and intellectual disabilities.

Core communication board, example of a high-frequency vocabulary system commonly used in New Zealand AAC practice. The board combines core words for everyday communication with fringe vocabulary tailored to specific activities or contexts.

Part 3
The Shape of the Proposal

Focus	PictoNet Meaning through Images
The challenge	Translate communicative intention—written as text—into a pictogram that is cognitively accessible, clear, and dignified.
Where the gap is	AAC still relies on illustrators and fixed libraries; systems do not yet generate pictograms directly from language.
What to design	A transformer-based LLM that converts natural language into SVG pictograms—using a visual grammar linking verbs, objects, and contexts.
Why it matters	Builds a bridge between text and image, expanding communication for people with complex communication needs.

PictoNet Pipeline → "Drawing-with-Thought" (DwT)

Step 1 Semantic Analysis (NLU)	Step 2 Conceptual Mapping	Step 3 Hybrid SVG Generation	Step 4 Accessibility Post-Processing	Final Pictogram Output
"I want to drink water"
Converts text into structured meaning through speech-act detection, frame semantics, and NSM decomposition.	Maps semantic roles to visual symbols using a pictogram library and established graphical conventions. This is were the blending happens.	Generates an SVG scaffold via LLM reasoning, refined by a diffusion model for visual coherence.	Adds accessibility metadata (ARIA, WCAG) to ensure compatibility with assistive technologies.	Integrates user feedback through evaluation and RLHF fine-tuning for iterative model improvement.

Semantic analysis — NLU front-end

Please make your bed


{
"utterance": "Please make your bed",
"lang": "en",
"metadata": {
  "speech_act": "directive",
  "intent": "request"
},
"frames": [
  {
    "id": "f1",
    "frame_name": "Directed_action",
    "lexical_unit": "make",
    "roles": {
      "Agent": {
        "type": "Addressee",
        "ref": "you",
        "surface": "your"
      },
      "Theme": {
        "type": "Object",
        "lemma": "bed",
        "surface": "bed",
        "definiteness": "definite"
      }
    }
  }
],
"logical_form": {
  "event": "make(you, bed)"
},
"pragmatics": {
  "politeness": "polite",
  "formality": "neutral",
  "expected_response": "compliance"
},
"visual_guidelines": {
  "focus_actor": "you",
  "action_core": "make",
  "object_core": "bed",
  "context": "bedroom",
  "temporal": "immediate"
}
}

Focus	PictoForge Designing with AI
The challenge	Treat AI as a design material—directly manipulable, refineable, and understandable through use.
Where the gap is	Most generative tools hide inner logic; users cannot see how outputs are built or teach the system how to improve.
What to design	An interactive interface where users adjust pictograms and each correction becomes structured feedback for retraining (direct manipulation & RLHF).
Why it matters	Turns machine learning into a visible, human process—understanding grows through collaboration, not opacity.

Focus	MediaFranca Language as a Commons
The challenge	Promote fair use of data for AI training, respecting authorship, privacy, cultural identity, and model portability.
Where the gap is	Centralised infrastructures concentrate data and power; communities lose control over voices, languages, and creative work.
What to design	A federated, open-source platform where local instances train their models and share improvements responsibly (cultural sovereignty & ethical governance).
Why it matters	Redefines AI as a collective resource rather than a corporate asset—communities own, adapt, and evolve their languages.

Thank you

The MediaFranca Initiative is an open, collaborative framework for pictographic AAC. It is in an early research and prototyping phase. Its focus is a transparent text-to-SVG generative pipeline: starting from a communicative intent, the system analyses the utterance (NLU), maps it to semantic structures, and outputs an editable, standards-compliant SVG pictogram.

The goal is accessible tooling and reproducible methods for researchers, practitioners, and communities. MediaFranca couples formal modelling (language and pictographic structure) with situated practice (evaluation with AAC stakeholders).

Core repositories in this foundational ecosystem:

Manifesto — ethical, social, and design principles for the ecosystem.
PictoNet — the semantic communication network that models meaning with pictograms and linguistic structures.
PictoForge — pictogram editor and RLHF interface. See the functional mock-up.
nlu-schema — natural language understanding schema for decomposing utterances into semantic and pragmatic structures (input to pictographic generation).
VCSCI — Visual Communicability and Semantic Correspondence Index for assessing the communicative adequacy of machine-generated pictograms.

Herbert Spencer González · PhD in Design (AUT)

herbertspencer.net/cc · herbert.spencer@autuni.ac.nz