devlog 2024-05-24
Wow, been a long time since my last devlog. It's been a weird few weeks, with some misc poor moods, lots of evening/weekend events. My actual job at Stanford has also been a lot recently.
Anyway – I thought I'd do a whole-ass overview of what's been going on and what's changed.
Product
I sat down and figured out the entire API the server needs to provide, what tools the front-end needs to be able to handle, and basically how all of comind is supposed to work as a product. I'll be writing this up over the next few days thought it will sadly be internal for the moment.
It'll be nice to sit down and work through each of the API points. There are a handful of people in a Telegram channel with me who are vaguely interested in the project – some of them have offered to help, but I have been extraordinarily bad at project management and making Comind a hospitable environment for new contributors.
Hopefully, building a clear, concise design document for the backend will help me get other folks involved to the extent that they are interested.
The product for comind is a server and an API, and the company will provide an open-source front-end for public use and development. Later on, we'll provide a paid-for commercial front-end that's not dissimilar from Slack on growth hormones.
I am occasionally annoyed at how long it has taken me to appreciate the benefit of completing a complete plan for a product. I don't think I could have done anything different, since all my fucking around thusfar has been extremely helpful for me understanding exactly what I want comind to be.
Time to get serious and get shit out the door. This is a good product.
Backend
Lots of tinkering with graph database stuff and kind of just hating all the tools that exist. It will surprise nobody that the API support in Julia for various graph databases like Neo4j, TigerGraph, TerminusDB, etc. are either broken to shit or completely non-existent.
I ended up writing an entire Julia package for property graphs (PropertyGraphs.jl) that supports pattern matching queries, with the eventual goal of producing an industrial-grade graph system for Comind's internal usage. The work was heavily inspired by SQLiteGraph.jl, an incredibly simple package.
using PropertyGraphs
# Set up some nodes
john = Node(1, "person", "human"; name="John Doe", age=25)
jane = Node(2, "person", "human"; name="Jane Doe", age=26)
# Set up some edges
john_jane = Edge(1, 2, "friend of"; since=2010)
jane_john = Edge(2, 1, "friend of"; since=2011)
husband = Edge(1, 2, "husband of"; since=1992)
wife = Edge(2, 1, "wife of"; since=1992)
# Make the graph
graph = PropertyGraph(
Node[john, jane],
Edge[john_jane, jane_john, husband, wife]
)
# Perform a match query to get all "human" nodes,
# returns a new graph with john and jane.
match(graph, NodePattern("person", "human"))
# returns just john
match(graph, NodePattern(name="John Doe"))
# Matching edges is similar, this returns a graph with
# just the "friend of" edges
match(
graph,
EdgePattern("friend of")
)
# You can also provide an arbitrary number of patterns to
# match on. For example, this will return a graph with
# john and jane, as well as the husband/wife edges
# between them.
match(
graph,
NodePattern(name="John Doe"),
NodePattern(name="Jane Doe"),
EdgePattern("husband of"),
EdgePattern("wife of"),
)It was fun to write this, and it came together very quickly. It sadly does not implement the Graphs.jl abstract graph interface (yet) as there was some overhead I just wasn't willing to deal with for a prototype package.
Eventually, PropertyGraphs.jl will sit on top of postgres and will be easily loaded/saved to disk, and the comind server will essentially just be running a large, in-memory graph DB. Over time the plan is to add tooling for distributed graph processing.
Frontend
The frontend has come a long way. The design is simple and clean, though lacking a lot of features.
You can also talk to people on it in melds. My friend Grace and I had our first conversation on Comind a few days ago – it was broken as hell, you had to refresh the page to see new messages – but it worked. Fuckin incredible.
There are publicly accessible melds, the first of which is the void cafe:
I did a massive overhaul to how the front-end handles suggestions (thoughts you might want to think about). Now, there's a little "think" button in the bottom right that will show you thoughts that are related:
Pressing the think button currently does semantic search to show you related thoughts, but we're working on the recommender system to show you more relevant thoughts that include graph information, popularity, recency, etc.
Things I like about the design - It's simple and clean - It has a minimalistic design - It's unobtrusive
Things I don't like about the design - It takes up too much screen real estate - It's slow and doesn't feel "snappy" - It's not the most mobile friendly
Misc
Life has been very much in the way lately. I've been able to do some work on comind here and there but I've been quite burned out from work and I often feel like I need a break. The nights and weekends thing I've been doing is fun and all, but it has been very difficult to juggle having a job, trying to exercise regularly, and spending time with people I love.
At the same time, I'm also looking for work as my postdoctoral contract ends relatively soon. Sadly, I do not have the financial resources to take a vacation of any kind, otherwise I would happily take off several months to work on comind. The days when I have a full workday are absolutely insane – I end up accomplishing the same amount of work in a day that I do over an entire week of nights & weekends.
Anyway. Looking forward to more stuff. I am delighted with the property graph stuff, and I'm happy I was able to sit down and hand-write the entire business down. Feels good to have a plan.
– Cameron