Obsidian is a very cool markdown driven notetaking app. It's a very general purpose note taking app, and can do everything that every other note taking app can. It is also super extendable, which is really nice -- it is an electron desktop app, so plugins can be literally anything. There are plugins that add full on react UI components, and plugins that mess with webasm. Plugins are implemented by hooking onto type declarations that Obsidian provides, and it all is very well integrated into the app.
The coolest part of Obsidian though, in my opinion, is the fact that all of your notes live as a hierarchy of obsidian flavored markdown files (which is just a superset of markdown that adds things like highlighting, comments, and callouts). Obsidian itself is closed source, but I'm okay with using it anyway since my files are not obfuscated in any way.
A lot of people have extended obsidian with its powerful plugin plugin system, to let you do more than just take notes. One philosophy that seems to be catching on is to use Obsidian as a life manager; as an all in one tool.
Obsidian is Electron, so it can do anything a browser can. It's implemented very well and you don't really notice this unless you try, but if you want to create for Obsidian, the sky is the limit -- heck, you can create plugins with React.
One plugin I've started using is Obsidian Full Calendar, which a cool calendar plugin for Obsidian that lets you unidirectionally sync iCal calendars, and then also manage events within Obsidian, where it can treat events as markdown files. That means that I can create an event within Obsidian, and it can create a corresponding event markdown file, that has the actual metadata for the event in it.
This is a paradigm that I really like -- treating usually structured and rigid data as semi-structured markdown.
I have started to use templates to create other types of "typically structured dataforms" in unstructured markdown. The nice thing about Obsidian is that this does not mean copying and pasting every single template. Templating in Obsidian is pretty mature at this point, it comes with a built in templating plugin, and the community has created a much more powerful Templator plugin. You can plug in everything from the current note's file name, to the date, to arbitrary javascript function results.
I think one of the coolest Obsidian plugins, that really does totally change the experience, is Omnisearch, which is a super powerful search engine for your entire note vault. It does weighted fuzzy finding across all vault contents, and it even OCRs images and PDFs. What that means is that the search results are always as relevant as possible -- if you search for a file, files with some of that name (it's fuzzy!) show up first, followed by notes with titles, and contents. This means that if all of my life is stored as markdown in my Obsidian vault, then it's easy to find things later.
Part of this does mean that I'm restarting systems that I've had in place for a really long time. I really like Google Calendar, but I prefer being able to put whatever I want into semi structured markdown files for events. Part of switching to this new system means that my old calendar events are not going to be in my Obsidian vault.
Importing old calendar events is good enough for now, because since the plugin supports linking iCal I still can view all my old events, I just don't have the ability to have corresponding markdown files. I probably could figure out how to import them that way with some scripting, but that's not really worth it for me since those events are in the past and I'm probably not still taking notes on them.
One area of my note-taking life that I would like to include in my Obsidian vault, though, is my contacts. All my contacts before this project were, as was the case for most people, stored in Google Contacts, and on my phone in the Contacts app.
What I wanted was a system to load my contacts into structured markdown files (ideally following some sort of template that I could fine tune and specify). Obsidian has a plugin registry built into the app, which you can contribute to by just PRing to a repo. It is pretty extensive, but there were really limited options that showed up. It's possible there's some project out there on Github, since Obsidian plugins get loaded by just plopping a manifest and js bundle into a specific spot, but I couldn't find any.
But also, I want to get better at remembering people. I encounter a lot of people on a fairly regular basis, but I think I could do a better job of remembering what people do, interesting things I talk about with them, and that sort of thing.
By having all my contacts in raw markdown, I think it will both lower the barrier to someone being a "contact," and will allow me to keep better records of how I know people and track my conversations with them over time.
If I assume that all my contacts are in Google contacts (a bold proposition), then I can export them as a CSV and then use them that way. But that also means that anyone who uses my tool will also need to be using Google contacts. I guess I could have asked people to upload their contacts to Google so that they could then download them as a CSV, but that's clunky, so I decided to look into how they exist on the phone itself.
I can't really know how Apple is storing them on the file system because of how closed source it is, but one thing I did figure out is that Apple let's you export your contacts in a uniquely unintuitive way: press and hold on a contact "list", or "all contacts", and choose to Export (or email every single contact in your phone! I will not be doing that, though.).
It turns out that the format that gets exported is the vCard
format, which is the same format that gets used when you share contacts: if you text a friend your contact what you've sent them is a vCard
file.
vCards are kinda a pain. There's a proper RFC on the spec. It's a 74 page PDF. The vCard format was originally developed by Apple, AT&T, IBM, and Siemens. as a new unified standard for contact storage, but it's very powerful.
The US library of congress has a section for the format, so I'll drop in their description and move on
Virtual Card Format (vCard) is a versatile data format designed for exchanging electronic representations of contact information. vCard is commonly referred to as an "electronic business card" and is fully and openly standardized through IETF RFC 6350. The vCard file contains the same type of content typically found on a physical business card, such as a contact’s name, address, phone number and email but may also list more personal data such as birthday or even organizational data such as line of supervision. Being digital, vCards can contain graphics (including headshots or ID photos), video, and audio as well as textual data. These files are shared across a wide variety of communication channels including email, instant messaging, text messaging, and website embedding. A single vCard file can contain information for one or more contacts. These digital cards are versatile, extending beyond personal descriptions to potentially represent various directory objects, including organizations, departments, or even buildings. However, the predominant use remains the representation of individuals.
That's right, you can store arbitrary data in vCards! Pretty cool.
So, vCards are plain text that store contact information. Great, that means we can look at them (and diff them, and all the other fun things that plain text let's us do)...
Here's my vCard
1BEGIN:VCARD 2VERSION:3.0 3PRODID:-//Apple Inc.//iPhone OS 17.5.1//EN 4N:Mermelstein;Wolf;;; 5FN:Wolf Mermelstein 6X-PHONETIC-FIRST-NAME:Wolf 7EMAIL;type=INTERNET;type=pref:wsm32@case.edu 8item1.EMAIL;type=INTERNET:wolf.mermelstein@case.edu 9item1.X-ABLabel:University Alt 10item2.EMAIL;type=INTERNET:wsm32@cwru.edu 11item2.X-ABLabel:University 12EMAIL;type=INTERNET:uptothebird@gmail.com 13item3.X-ABLabel:Primary 14item5.TEL;type=pref:(917) XXX-7875 15item5.X-ABLabel:Backup 16TEL;type=CELL;type=VOICE:(929) XXX-7180 17NOTE:This is me. 18item6.URL;type=pref:https://404wolf.com 19item6.X-ABLabel:_$!<HomePage>!$_ 20BDAY:XXXX-REDACTED-08 21PHOTO;ENCODING=b;TYPE=JPEG:/9j/4AAQSkZJ....SIJ 22END:VCARD
I got it by clicking "share contact," and then I shared it via email with myself, and downloaded the email attachment.
It's pretty gnarly. I mean, it has all I need, but how do I actually yoink out the relevant pieces of information. Also notice that the image is just a regular base64 image -- that's pretty nice.
So I looked into it a bit.
It turns out that there's not a ton of good options. I tried three different ones before arriving on vcf, and faced a few issues:
So, vcf
isn't typed, but it does have type declarations, and works reasonably well. And, it turns out it's wicked fast with bun
, which is really nice. I don't even have to bundle, just running it as a script is fast with bun
(but we'll bundle later since I do need to package it with nix, since of course we do).
1vCard { 2 version: "3.0", 3 data: { 4 version: Property { 5 _field: "version", 6 _data: "3.0", 7 is: [Function: is], 8 isEmpty: [Function: isEmpty], 9 clone: [Function: clone], 10 toString: [Function: toString], 11 valueOf: [Function: valueOf], 12 toJSON: [Function: toJSON], 13 }, 14 prodid: Property { 15 _field: "prodid", 16 _data: "-//Apple Inc.//iPhone OS 17.5.1//EN", 17 is: [Function: is], 18 isEmpty: [Function: isEmpty], 19 clone: [Function: clone], 20 toString: [Function: toString], 21 valueOf: [Function: valueOf], 22 toJSON: [Function: toJSON], 23 }, 24 n: Property { 25 _field: "n", 26 _data: "Mermelstein;Wolf;;;", 27 is: [Function: is], 28 isEmpty: [Function: isEmpty], 29 clone: [Function: clone], 30 toString: [Function: toString], 31 valueOf: [Function: valueOf], 32 toJSON: [Function: toJSON], 33 }, 34 fn: Property { 35 _field: "fn", 36 _data: "Wolf Mermelstein", 37 is: [Function: is], 38 isEmpty: [Function: isEmpty], 39 clone: [Function: clone], 40 toString: [Function: toString], 41 valueOf: [Function: valueOf], 42 toJSON: [Function: toJSON], 43 }, 44 xPhoneticFirstName: Property { 45 _field: "xPhoneticFirstName", 46 _data: "Wolf", 47 is: [Function: is], 48 isEmpty: [Function: isEmpty], 49 clone: [Function: clone], 50 toString: [Function: toString], 51 valueOf: [Function: valueOf], 52 toJSON: [Function: toJSON], 53 }, 54 email: [ 55 [Object ...], [Object ...], [Object ...], [Object ...] 56 ], 57 xAbLabel: [ 58 [Object ...], [Object ...], [Object ...], [Object ...], [Object ...] 59 ], 60 tel: [ 61 [Object ...], [Object ...] 62 ], 63 note: Property { 64 _field: "note", 65 _data: "This is me.", 66 is: [Function: is], 67 isEmpty: [Function: isEmpty], 68 clone: [Function: clone], 69 toString: [Function: toString], 70 valueOf: [Function: valueOf], 71 toJSON: [Function: toJSON], 72 }, 73 url: Property { 74 group: "item6", 75 type: "pref", 76 _field: "url", 77 _data: "https://404wolf.com", 78 is: [Function: is], 79 isEmpty: [Function: isEmpty], 80 clone: [Function: clone], 81 toString: [Function: toString], 82 valueOf: [Function: valueOf], 83 toJSON: [Function: toJSON], 84 }, 85 bday: Property { 86 _field: "bday", 87 _data: "XXXX-REDACTED-08", 88 is: [Function: is], 89 isEmpty: [Function: isEmpty], 90 clone: [Function: clone], 91 toString: [Function: toString], 92 valueOf: [Function: valueOf], 93 toJSON: [Function: toJSON], 94 }, 95 photo: Property { 96 encoding: "b", 97 type: "jpeg", 98 _field: "photo", 99 _data: "/9j/4An/1/...........S/+gpXY19Bc8RrU/9k=", 100 is: [Function: is], 101 isEmpty: [Function: isEmpty], 102 clone: [Function: clone], 103 toString: [Function: toString], 104 valueOf: [Function: valueOf], 105 toJSON: [Function: toJSON], 106 }, 107 }, 108 get: [Function: get], 109 set: [Function: set], 110 add: [Function: add], 111 setProperty: [Function: setProperty], 112 addProperty: [Function: addProperty], 113 parse: [Function: parse], 114 toString: [Function: toString], 115 toJCard: [Function: toJCard], 116 toJSON: [Function: toJSON], 117}
So we have what we need, and we can interface with it with typescript
now. It seems everything that we need is there, but it's just a pain in the neck to extract anything.
some hours later
I decided to move on anyway, and write a wrapper that uses vcf
to extract the necessary pieces of information from the vCard
to craft a Contact
object.
1{ 2 name: { 3 first: "Wolf", 4 last: "Mermelstein", 5 pronunciation: "Wolf", 6 }, 7 organization: "", 8 title: "", 9 phones: [ 10 { 11 number: "(917) XXX-7875", 12 type: "Backup", 13 }, { 14 number: "(929) XXX-7180", 15 type: "Misc", 16 } 17 ], 18 emails: [ 19 { 20 address: "wsm32@case.edu", 21 type: "Misc", 22 }, { 23 address: "wolf.mermelstein@case.edu", 24 type: "University Alt", 25 }, { 26 address: "wsm32@cwru.edu", 27 type: "University", 28 }, { 29 address: "uptothebird@gmail.com", 30 type: "Misc", 31 } 32 ], 33 addresses: [], 34 websites: [ 35 { 36 url: "https://404wolf.com", 37 label: "HomePage", 38 } 39 ], 40 birthday: "undefined 8, XXXX", 41 notes: "This is me.", 42 image: { 43 data: "/9j/4AAQSkrwZ/x..........XY19Bc8RrU/9k=", 44 type: "jpeg", 45 }, 46}
Much, much better.
Most of the pain was in dealing with how Apple hacks about the vCard
spec: for many of the elements in the parsed response I got, I would find that things like phone numbers would be labeled with tags like label1
, label2
, and label3
, corresponding to personal
, home
, etc, where the mapping would exist elsewhere in the vCard
, but not in one consistent spot.
To solve this, I just created a global hashmap for each vCard
that had all of the label references and their corresponding proper labels...
1const rawLabels = this.getPropertyJSONs("xAbLabel"); 2let labelMap = []; 3if (rawLabels !== null) { 4 labelMap = Object.fromEntries( 5 rawLabels.map((label: any) => [ 6 (label[1] as any).group, 7 sanitizeVCardLabel(label[3]).trim(), 8 ]), 9 ); 10} 11const getLabel = (label: string) => { 12 if (label in labelMap) return labelMap[label]; 13 else return labelFallback; 14};
Now that I have a function that can take a contact and give me a pretty JSON, I'm just about ready to start generating markdown.
For this, I have a few options, but it was a tricky decision.
One cool idea I had pretty early on for this project was the ability for the process to be bidirectional. That is, I thought it'd be pretty neat if you could take vCard contacts and turn them into markdown, and then eventually go the other direction: take the markdown you generated, and maybe some hand written contacts that follow the schema, and create vCards that you can load back into your phone.
Also, it would be important to parse the generated markdown later on just so that I could change the template if I ever wanted to. I love configuring things, and I think it's pretty likely that I would want to make some changes to the template down the line, like moving different sections of the contacts to different spots in the template.
No matter what I did, I'd probably be using remark to parse the markdown, which is a plugin for the unified
AST ecosystem. It lets you create a traversable AST from a plaintext markdown file. I was thinking it might be fun in the future to create my own vCard parser using unified
in pure typescript
.
This was a really tough decision, I was brainstorming and devised a few different options that would be potentially bidirectional
It turns out that markdown allows you to inject something called "frontmatter", which is basically a yaml, at the top of your markdown file, surrounded by ---
on either side. That is, something like
---
foo: "bar"
tags: foo, bar, buzz
---
This is something I do for the Obsidian plugin for my website; I embed metadata about the blog posts at the top of the blog post files.
One idea then, using this, would be to pick a template and stick to it, and write a parser that can parse it into a specific format. Then I could add something like "parser-top-use" (in more practicality, probably something like "version" or "parserid"), and then dynamically parse notes based on the schema that they are using.
This is a pretty high effort idea, so another idea I was flirting with was rendering markdown based on the frontmatter. I found a very straightforward Obsidian plugin that does this called "Obsidian handlebars". Handlebars is a javascript library that lets you compile and then use templates for text files. It's a templating language and is pretty good at what it does.
The plugin's example is something very similar to what I want to do...
1--- 2tags: 3 - cool 4 - awesome 5--- 6 7\`\`\`handlebars 8tags: {{#each frontmatter.tags}}{{.}}, {{/each}} 9\`\`\`
Unfortunately, however, it's one directional.
I found an Obsidian plugin called "Obsidian meta bind" that does something like this in a bidirectional way.
You can create "inputs" in the markdown template, and then when you edit them they automatically update the frontmatter, and then when the user edits the inputs OR the frontmatter it automatically updates the note/frontmatter.
It works pretty good, but it's fairly janky and requires a lot of boilerplate that I rather not deal with. It also seems a bit fragile, since it is assuming that their inputs work properly long term and don't have any weird plugin interactions.
Another idea that I had was to very simply just create a vCard viewer/editor for Obsidian. I've wanted to learn Svelt for a while, so I might still do this eventually, but for now, I decided on a simpler solution.
Obsidian tables are pretty good for storing tabulated data, and they are very human readable. It's an accessible format, and remark/unified should be able to parse it easily.
1| This is a header | This is another header | 2| ---------------- | ---------------------- | 3| This is a value | This is another value |
I decided to go with storing the contact data in tables for now, since it'll be easy to parse them later on. To figure out which sections are which, for now I'll just have headings above the tables that say what is in them, that I can regex against when parsing. It is a bit restrictive to require that people structure their contact templates with markdown tables, but it's better than zero flexability, and it will let me figure out a better solution later on.
The example template that I came up with, which is very table driven to store the key-value pairs, is this.
1--- 2obsidian-contact-importer-schema-version: "1.0" 3tags: ["person"] 4--- 5Imported with [Obsidian Markdown Importer](https://github.com/404Wolf/obsidian-contact-importer) 6 7--- 8 9{{#if organization}} 10 {{#if title}} 11{{title}} @ {{organization}} 12 {{else}} 13{{organization}} 14 {{/if}} 15{{else}} 16 {{#if title}} 17{{title}} 18 {{/if}} 19{{/if}} 20{{#if image}} 21![Image]({{image}}) 22 23{{/if}} 24## Phones 25| Type | Number | 26|:--------------|:--------------| 27{{#each phones}} 28| {{type}} | `c!{{number}}`| 29{{/each}} 30 31## Emails 32| Type | Address | 33|:--------------|:--------------| 34{{#each emails}} 35| {{type}} | `c!{{address}}` | 36{{/each}} 37 38## Socials 39| Type | Handle | 40|:--------------|:--------------| 41 42## Links 43| Type | URL | 44|:--------------|:--------------| 45{{#each websites}} 46| {{label}} | `g!{{url}}` | 47{{/each}} 48 49## Addresses 50| Type | Street | City, State | Zip, Country | 51|:--------------|:--------------|:--------------|:--------------| 52{{#each addresses}} 53| {{type}} | {{street}} | {{city}}, {{state}} | {{zip}} {{country}} | 54{{/each}} 55 56## Other 57| Type | Value | 58|:--------------|:--------------| 59{{#if birthday}} 60| Birthday | `g!{{birthday}}` | 61{{/if}} 62| Imported | {{imported.month}} {{imported.day}}, {{imported.year}} | 63 64--- 65 66{{notes}}
Notice that I'm using handlebars to finally slot the json values into the template. To make this future proof and allow people to modify it to their liking, instead of remapping all of the handlebar fields in the templator function, I'm just dynamically dumping the JSON into the handlebar...
1export default async function templateMarkdown( 2 contact: ContactType, 3 markdownTemplate: string, 4) { 5 const template = handlebars.compile(markdownTemplate); 6 const now = new Date(); 7 return template({ 8 ...contact, // Dump the entire contact json as-is 9 image: 10 contact.image === null 11 ? null 12 : `${(await getB64Hash(contact.image.data)).slice(0, 16)}.${contact.image.type}`, 13 imported: { 14 month: getFullMonthName(new Date()), 15 day: now.getDate(), 16 year: now.getFullYear(), 17 }, 18 }); 19}
And it works! So now let's package it and call it a day.
I'm not going to go too deep into Nix here, but the TLDR is that it's a programming langauge and utility library that makes it very easy to create extremely pure and consistent, builds of arbitrary files, including executable.
The catch with nix
is that it doesn't just try to be pure, it forces purity. Purity is, in this context, that whenever you run a build, the output artifact will be bit for bit identical given the same inputs. This makes it annoying to package things some times, since the sandbox
that nix
builds happen in is very restrictive -- it doesn't have internet!
There's a bunch of utility builder functions that are able to prefetch all your dependencies. A common one is buildNpmPackage
, which is de facto standard for nixpkgs
packages (packages on the official registry) that are in ts/js
. I'm using bun
though, not npm
or node
.
Prefetching artifacts for builds is done by specifying the hash of the thing you're fetching, so that you can guarantee that the things you're fetching are what you think they are. This isn't that bad when you have a package-lock.json
with npm
or even with yarn
. In that case all of the hashes already live there, and can be used to safely do the fetches.
But, in this case, I want to use nix
to package a bun
app. bun
presents a few challenges; namely, it uses a special binary lock file format that's super fast and efficient, but isn't easily parseable. You can use bun
to read it as a yarn
lock file, and theoretically you could go down that route to do prefetching of deps, but I decided on a much easier approach: fixed-point-derivations.
Fixed point derivations are when you ALLOW internet access in the sandbox during the build. This sounds scary, and like a bad idea. It kinda is. The catch is that you specify the hash of the output before doing the build. If the build, using the internet during the build, results in an output with the same hash that you pre-specified, then the build succeeds an it's fine, but otherwise the build fails and you get nothing. In the case of bun
though, this isn't that bad, since bun
is very good at getting very consistent dependencies using its lock file. So it's probably good enough for now.
Here's what it looks like in action, as a DIY nix builder utility function...
1{ 2 pkgs, 3 src, 4 name, 5 bun ? pkgs.bun, 6 buildCommand ? "build", 7 outputHash, 8 outputHashAlgo ? "sha256", 9 outputHashMode ? "recursive", 10 ... 11}: 12pkgs.stdenv.mkDerivation { 13 inherit 14 name 15 src 16 outputHash 17 outputHashMode 18 outputHashAlgo 19 ; 20 buildInputs = [ bun ]; 21 buildPhase = # bash 22 '' 23 bun install 24 bun run ${buildCommand} 25 ''; 26 installPhase = # bash 27 '' 28 mkdir -p $out/bin; 29 cp -r ./dist/* $out/bin; 30 ''; 31}
Basically, do the bun install
in the sandbox, do the bundling in the sandbox, and then move the output to an output folder. To make it executable, I just wrap it in a shell script...
1default = pkgs.writeShellScriptBin "obsidian-contact-importer" '' 2 ${pkgs.bun}/bin/bun run ${bun-utils.lib.${system}.buildBunPackage { 3 src = ./.; 4 name = "obsidian-contact-importer"; 5 outputHash = "sha256-H9hWffy5QUN/n9tgaOO51k92XPJyLQ/bneFRgseCiX0="; 6 }}/bin/index.js 7'';
Which just runs bun on the output file. Javascript isn't "compiled", so this is the best we're going to get. What this does do though is create a nice bundle, and makes the build nicely portable. It also means that anyone with nix can instantly run the project with a single CLI command (see the next section)!
Clone this repo, or create a inputs
folder with a vcards.vcf
and template.md
file (see example template if you're creating your own).
Run nix run github:404wolf/obsidian-contact-importer
if you have nix
. If not, make sure that bun
is installed, and then run bun install
. Export your contacts to a .vcf
format, and then just run bun run dev
.
Another super cool Obsidian plugin, so cool that it's even baked into the program, is "Daily Notes". Daily notes are exactly what they sound like -- Obsidian automatically creates a new note every day that you can dump information into, and by virtue of the structure they get sorted by date. I find it really useful for very quickly jotting things down, and since they're in my vault they're nicely searchable.
A big inspiration for the project is making it easier to take notes on people, so the ability to make people reference-able in my vault with a single hotkey would rapidly reduce the friction needed to begin taking notes on a person. Since I'm used to pinging people on apps like Slack, Zulip, Discord, etc, I thought it'd be handy if I could just @ my contacts in my vault.
Enter @ symbol linking..
An important part of Obsidian is the ability to easily link together your notes. Obsidian lets you reference any note in your vault by using [[Note Title]]
, and it creates a clickable link. You can add a ! to cause the reference to actually show its contents in the current note.
@ Symbol linking wasn't actually the first plugin I found for this purpose, though. I started off using Obsidian at people, which claims to do literally exactly what I want. Unfortunately, even though there's a setting for it, all of the people notes you create with it end up in the top level directory of your Obsidian vault, when I want them to end up in a /People
folder.
@ Symbol linking is a pretty straightforward plugin that does the same thing. It's a bit more generalized -- you can use other symbols as prefixes besides @, and you can customize a few more options.
One thing that I did like that Obsidian at people
did was prefixing all of my people notes with "@", though, which makes it really clear which notes are "People" notes. This was a feature lacking in @ Symbol linking so Idecided to implement it myself and PR the change.
Their plugin was written pretty well, though more comments and docs would've been nice. It mostly boiled down to adding
1if (settings.keepTriggerSymbol) 2 filePath = value.obj?.filePath.replace(/([^/]+)$/, `${settings.triggerSymbol}$1`); 3else 4 filePath = value.obj?.filePath;
And the relevant setting. To polish things off, I also updated the "Create new note" suggestion to suggest the name with the @ symbol if the setting was enabled.
To make the templates look nicer, I decided to use a plugin I already was using in my vault for my contacts too. Obsidian inline admonitions lets you change the way that your inline code (` <this type of thing> `) renders so that it can have a nice color and even be slightly translucent. I decided to change the template slightly to incorporate these.