I've been vaguely working on a side project for Comind, colang, that's kind of one of the distant-future things I want on the platform.
colang is a domain-specific language (DSL) built in Julia. It's intended to provide a robust interface for building massive-scale language programs. A language program is a series of natural-language processes that perform aribtrarily complicated tasks, such as building a documentation website, retrieving a strongly-typed list of bullet point items, etc.
Something I've found myself needing a lot is a sequence of prompts for a language model, but structured to make it easy to work on very large language programs. There are tools like this, like LangChain or DSPy. These are both great tools and you should use them, but they are not a first-class Julia experience and they probably won't be.
So, I'm making my own, and I'm going to experiment with some design decisions. colang is intended to take a functional and strongly-typed approach to building language programs – you should know at the language level whether you are getting a Yes, a No, or a Maybe when you ask a question. Those are types that you can dispatch to different types of program.
Here's an overview of my work this evening on colang.
The goal of colang is to provide a way to build and manage a sequence of prompts for a language model, but structured to make it easy to work on very large language programs.
Here's a small program I could imagine wanting to run:
# Get only non-empty thoughts
remove_empty(thoughts) = filter(t -> !(t isa Empty), thoughts)
# Extract tasks from the input text. meta_memory is memory
# passed into the function from the caller, but extract_tasks
# chooses not to use it.
function extract_tasks(meta_memory::AbstractMemory, input_thought)
# Memory for this
memory = Stack(
# Core thought
Thought("I am responsible for extracting tasks from the input text."),
# Thoughts to use for context
ThoughtArray()
)
# Ask if this input text contains at least one task.
# If not, return Empty.
if !contains_task(memory, input_thought)
return Empty()
end
# If it does, return a list of tasks.
tasks = TaskThought[]
while true
# Try to get the next task
task = get_first_task(memory, input_thought)
# If it's empty, we're done.
if task isa Empty
break
end
# Add the task to the memory, noting that we've already
# processed it. ProcessedTask here would be reduced in the prompt
# to say "Already processed this, do not duplicate: " or something
# similar.
push!(memory.thoughts.thoughts, ProcessedThought(task))
# Add the task to the list of tasks
push!(tasks, task)
end
return remove_empty(tasks)
endHere, the extract_tasks function is a simple example of a voice (there will be explicit types later for dispatching). It takes in an input text and returns a list of tasks. The extract_tasks function returns a list of TaskThoughts, which can further be dispatched.
An example (that doesn't work with the current design):
tasks = extract_tasks(thought"I need to buy a dog, a horse, and some bourbon.")should return
TaskThought[
TaskThought("Buy a dog"),
TaskThought("Buy a horse"),
TaskThought("Buy some bourbon")
]Each of these of course could be used elsewhere. For example, if you are working on planning problems, you could do something like
memory = Stack(
# Core goal
Thought("""
I am responsible for planning a trip to the grocery store. I need
to plan out a series of tasks to get the items I need.
"""),
# Tasks
ThoughtArray([])
)
# Ask the language model to start planning
plan_thought = plan(
memory, """
What do I need to know to plan a trip to the grocery store?
"""
)
# Get the list of tasks from the plan
task_list = ask(
memory,
plan_thought
)
# Go through each task and ask the language model to perform it
for task in task_list
# do more stuff with the task
# . . .
endCurrently there are three abstract types, AbstractThought, AbstractMemory, and AbstractVoice. The AbstractVoice stuff isn't ready yet but I know it'll be part of the design.
#
# ████████╗██╗ ██╗██████╗ ███████╗███████╗
# ╚══██╔══╝╚██╗ ██╔╝██╔══██╗██╔════╝██╔════╝
# ██║ ╚████╔╝ ██████╔╝█████╗ ███████╗
# ██║ ╚██╔╝ ██╔═══╝ ██╔══╝ ╚════██║
# ██║ ██║ ██║ ███████╗███████║
# ╚═╝ ╚═╝ ╚═╝ ╚══════╝╚══════╝
#
# (abstract ones)
#
"""
An `AbstractThought` is a thought that is accepted or send by a voice.
"""
abstract type AbstractThought end
"""
An `AbstractMemory` is a memory that is something you should take into
account when you are thinking about during your process.
"""
abstract type AbstractMemory end
"""
An `AbstractVoice` is a function mapping an input thought to an output thought.
"""
abstract type AbstractVoice end#
# ████████╗██╗ ██╗ ██████╗ ██╗ ██╗ ██████╗ ██╗ ██╗████████╗
# ╚══██╔══╝██║ ██║██╔═══██╗██║ ██║██╔════╝ ██║ ██║╚══██╔══╝
# ██║ ███████║██║ ██║██║ ██║██║ ███╗███████║ ██║
# ██║ ██╔══██║██║ ██║██║ ██║██║ ██║██╔══██║ ██║
# ██║ ██║ ██║╚██████╔╝╚██████╔╝╚██████╔╝██║ ██║ ██║
# ╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝ ╚═╝
#
# Little bits of information.
#AbstractThought captures a "unit of information". This can be numbers, strings, images, URLs, etc. Each type of thought has its own subtype. Some examples:
Thought is a string but has no special information. These can be dispatched in standard Julia.
Empty is when a thought is empty. The voice was not able to perform the task, or the task was not possible. This is akin to a "stop thought", because usually an Empty implies that a voice chain can be terminated.
NumberThought contains any numeric type, which can be useful for numeric operations that a voice may wish to perform.
AbstractCategoricalThought contains a few subtypes that are useful for categorizing things:
"""
`AbstractCategoricalThought` is a thought that can be categorized.
"""
abstract type AbstractCategoricalThought <: AbstractThought end
"""
YesNoMaybe is a categorical type that can be used to represent
yes, no, or maybe.
"""
abstract type YesNoMaybe <: AbstractCategoricalThought end
struct Yes <: YesNoMaybe end
struct No <: YesNoMaybe end
struct Maybe <: YesNoMaybe endYes, No, and Maybe are the most common subtypes of AbstractCategoricalThought, and are intended to make it easy to work with cases where you ask the language model:
User: Is water blue?
AI: Yes, water is usually blue.
In this case we want to determine that this is in fact probably a Yes, which can either be done via structured text extraction or by simple feedback-loops (i.e. "sorry that is not yes/no, please respond only with "yes" or "no".)
Thoughts are intended to be passed up and down the train of thought. Think of them as variables or values in a standard programming language – this one just happens to be an attempt to extract a bullet-pointed list.
I also included some memory types which are useful for managing the context of a language model call. RAG (retrieval augmented generation) is basically the best way to get performance out of your lagnuage models, and so I want different tasks to be able to share relevant information with each other.
#
# ███╗ ███╗███████╗███╗ ███╗ ██████╗ ██████╗ ██╗ ██╗
# ████╗ ████║██╔════╝████╗ ████║██╔═══██╗██╔══██╗╚██╗ ██╔╝
# ██╔████╔██║█████╗ ██╔████╔██║██║ ██║██████╔╝ ╚████╔╝
# ██║╚██╔╝██║██╔══╝ ██║╚██╔╝██║██║ ██║██╔══██╗ ╚██╔╝
# ██║ ╚═╝ ██║███████╗██║ ╚═╝ ██║╚██████╔╝██║ ██║ ██║
# ╚═╝ ╚═╝╚══════╝╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝ ╚═╝
#
# Everything you need to manage what you know.
#
"""
`BlankSlate` is a memory that is empty. No information can be added
or removed to it. Please create a [`Stack`](@ref) if you need to
store information.
"""
struct BlankSlate <: AbstractMemory endBlankSlate is the most common memory type, and is the default memory type for a language model call. It is empty, and cannot be used for storage. Passing BlankSlate to a voice means that it has no context and can only respond by itself.
An interesting thing to explore in language programming in general is how to allow voices and thoughts to share context and pass messages between them while making sure that certain voices are not permitted to access information that they should not have.
As a motivating example, consider a voice chain A -> B -> C. A has passed information to B that is not relevant for C. We don't want C to see the information because it may confuse C, or, in the case of sensitive information, contaminate the chain with knowledge of something we did not want it to know.
An early step there is BlankSlate – if B receives information from A and passes a BlankSlate to C, then C will effectively be sandboxed to an independet process.
I also made a Stack, which is a more flexible, mutable memory structure. A Stack is composed of two parts:
A core thought, which is the most important part of the Stack. This is the thought that will be passed to the language model when it is asked to perform a task. core is intended to be used when dictating the entire flow of a program or local memory space (defined as AbstractMemory shared by voices in the memory space) by describing the top-level task so that the voice can tailor its response to the ultimate goal of a memory space.
A thoughts array, which is the stack of thoughts. This is the actual data that the language model will be working with. Voices can add and remove thoughts to this array as needed as a way of passing messages to one another, or to provide general context to all voices in the memory space
"""
`ThoughtArray` is a vector of thoughts.
"""
mutable struct ThoughtArray{A<:Vector{AbstractThought}} <: AbstractMemory
thoughts::A
end
"""
A `Stack` memory has a vector of thoughts. You can push thoughts onto the stack
and pop thoughts from the stack.
There is an optional `core` thought that is used to guide all processes,
in the memory. All users of a `Stack` memory will have `core` prepended
to the final prompts.
"""
mutable struct Stack{A,B<:ThoughtArray} <: AbstractMemory
core::A
thoughts::B
endI'm enjoying the process. Good to take a break from the nuts-and-bolts of comind and work on something kind of goofy and fun.
– cameron