Why TongFlow starts with modalities

You have a document. You want to extract its text, make a summary and produce a version you can listen to. The task moves between files, text and audio, but the intention stays the same: make the information useful in another form.

That is where TongFlow starts. What do you have, what do you need, and which steps connect the two?

We call this approach modality-first. Materials have a form, capabilities connect those forms, and plugins and models carry out the work.

Three decisions, kept separate

A modality is a form of information: text, an image, audio, video or a 3D model. On the canvas, these become concrete materials you can work with.

Each material has its own node. The operations connected to it determine what happens next. Text can feed speech synthesis or image generation. An image can be interpreted, processed or used to generate video.

Building a workflow involves three distinct decisions:

• Material: which text, image or audio should this step use?
• Capability: what should happen to it—extraction, generation, transformation, combination or splitting?
• Implementation: which compatible plugin, and the model it provides, should do the work?

This changes the order in which you can make decisions. You can put “turn this text into speech” into the workflow before choosing its implementation. The model matters, but choosing it does not have to determine the structure of the entire task.

A result is something you can use again

Take the document example. One possible workflow looks like this:

Document → Extract text → Text
                           ├→ Summarize → Summary text
                           └→ Speech synthesis → Audio

Both branches use the same extracted text and keep their own outputs. If you later want to narrate the summary, you can connect speech synthesis to the summary text instead.

The useful part is that an output remains a material. An uploaded image and a generated image can both feed an operation that accepts images. The new audio can be processed further, or combined with a character image to make a talking video.

You do not have to design the whole workflow in advance. Build a section, inspect its results, then branch or combine from there. Work you have already done gives you a place to continue.

The same structure can serve different tasks

Image and video generation make the idea easy to see. Document extraction, transcription, content understanding and organizing materials fit the same structure.

A video, for example, can be split into clips. Their audio can be extracted and transcribed. The resulting text can then be organized or used in another generation step. Splitting one input into several outputs and moving between modalities are both parts of the workflow.

We think of TongFlow as a tool for building things. That might mean making a piece of media, developing a way to process information, or putting together a workflow you will use repeatedly. The task comes from you.

Add, transform, combine, split and batch are the basic operations. Much of their value comes from how they connect.

Keep the workflow when you change the model

Suppose one step generates an image. You start with plugin A, then want to try another compatible implementation.

TongFlow records the capability separately from the plugin that executes it. When an alternative meets the same input and output contract, you can keep the surrounding workflow and replace that step’s implementation. Its configuration, credentials and results still need checking.

In the code, capability contracts and the node registry make this separation explicit. The contract describes what goes into an operation and what comes out; the workflow specifies which plugin runs it. Your way of organizing the task can outlast a particular model choice.

Clear interfaces make composition possible

Modality-first does not make every node compatible with every other node. An operation that expects audio needs an audio input. One that combines an image and a voice needs both materials.

What you can run today depends on the available capabilities, compatible installed plugins and their configuration. Adding a modality or capability also requires a definition and an implementation.

The design provides a way to extend the system. Each compatible capability can give existing materials another route and combine with operations already available. That is the possibility we care about.

A workflow people and agents can both edit

The canvas offers a visual way to build, while the underlying structure can also be manipulated through code. The canvas, connection checks and exporter share a node registry. The DSH integration lets an agent build workflow files that you can open, inspect and edit on the canvas.

The handoff can therefore include the process as well as its output. If an input, capability or plugin needs changing, there is a specific step to change.

We want TongFlow to leave room for that next decision. Connect different modalities into a workflow, then keep building from what you have made.

Open TongFlow and start with a material you already have. For the implementation, see the source repository, capability registry and DSH integration.