Augmented Markdown Compiler • Marqua

Introduction

Marqua is an enhanced markdown compiler with code syntax highlighting and built-in front matter parser that splits your markdown into two parts, content and metadata. The generated output is highly adaptable to be used with any framework and designs of your choice as it is just JSON.

The markdown compiler is powered by markdown-it and code syntax highlighter is powered by Shiki.

The front matter parser for the metadata is powered by a lightweight implementation in-house, which supports a minimal subset of YAML syntax and can be used as a standalone module.

Quick Start

	
		
			
			
		
	

	pnpm install marqua

Use the functions from the FileSystem module to compile a file or traverse directories.

	
		
			
			
		
	

	import { compile, traverse } from 'marqua/fs';

compile(/* string */, /* optional hydrate callback */);
traverse(/* options */, /* optional hydrate callback */);

Add interactivity to the code blocks with hydrate from /browser module.

	
		
			
			
		
	

	<script>
	import { hydrate } from 'marqua/browser';
</script>

<main use:hydrate>
	<!-- content here -->
</main>

Getting Started

	
		
			
			
		
	

	pnpm install marqua

Include base styles

Make sure to include the stylesheets from /styles to your app

	
		
			
			
		
	

	<script>
	// process with JS bundler
	import 'marqua/styles/code.css';
</script>

<!-- choose one but not both -->

<style>
	/* process with CSS bundler */
	@import 'marqua/styles/code.css';
</style>

The following CSS variables are made available and can be modified as needed

	
		
			
			
		
	

	:root {
	--font-default: 'Rubik', 'Ubuntu', 'Helvetica Neue', sans-serif;
	--font-heading: 'Karla', sans-serif;
	--font-monospace: 'Fira Code', 'Inconsolata', 'Consolas', monospace;

	--mrq-rounding: 0.3rem;
	--mrq-tab-size: 2;

	--mrq-primary: #0070bb;
	--mrq-bg-dark: #2d2d2d;
	--mrq-bg-light: #f7f7f7;
	--mrq-cl-dark: #242424;
	--mrq-cl-light: #dadada;
}

.mrq[data-mrq='block'],
.mrq[data-mrq='header'],
.mrq[data-mrq='pre'] {
	--mrq-pre-bg: #525252;
	--mrq-bounce: 10rem;
	--mrq-tms: 100ms;
	--mrq-tfn: cubic-bezier(0.6, -0.28, 0.735, 0.045);
}

.mrq[data-mrq='header'] {
	--mrq-hbg-dark: #323330;
	--mrq-hbg-light: #feefe8;
}

Semantics

Front Matter

Marqua supports a minimal subset of YAML syntax for the front matter, which is semantically placed at the start of the file between two --- lines, and it will be parsed as a JSON object.

All values will be attempted to be parsed into the supported types, which are null, true, and false. Any other values will go through the following checks and the first one to pass will be used.

Comments, #; indicated by a hash followed by the value, will be omitted from the output
Literal Block, |; indicated by a pipe followed by a newline and the value, will be parsed as multi-line string
Inline Array, [x, y, 2]; indicated by comma-separated values surrounded by square brackets, can only be primitives
Sequence, - x; indicated by a dash followed by a space and the value, this can contain nested maps and sequences

To have a line be parsed as-is, simply wrap the value with single or double quotes.

	
		
			
			
		
	

	---
title: My First Blog Post, Hello World!
description: Welcome to my first post.
tags: [blog, life, coding]
date:published: 2021-04-01
date:updated: 2021-04-13

# do not assign top-level data when using compressed nested properties syntax
# because this will overwrite previous 'date:published' and 'date:updated'
# date: ...
---

The above front matter will output the following JSON object…

	
		
			
			
		
	

	{
	"title": "My First Blog Post, Hello World!",
	"description": "Welcome to my first post.",
	"tags": ["blog", "life", "coding"],
	"date": {
		"published": "2021-04-01",
		"updated": "2021-04-03"
	}
}

Where we usually use indentation to represent the start of a nested maps, we can additionally denote them using a compressed syntax by combining the properties into one key separated by a colon without space, such as key:x: value. This should only be declared at the top-level and not inside nested maps.

Content

Everything after front matter will be considered as content and will be parsed as markdown. You can use the !{} syntax to access the metadata from the front matter.

	
		
			
			
		
	

	---
title: "My Amazing Series: Second Coming"
tags: [blog, life, coding]
date:
  published: 2021-04-01
  updated: 2021-04-13
---

# the properties above will result to
#
# title = 'My Amazing Series: Second Coming'
# tags = ['blog', 'life', 'coding']
# date = {
#   published: '2021-04-01',
#   updated: '2021-04-13',
# }
#
# these can be accessed with !{}

# !{tags:0} - accessing tags array at index 0
This article's main topic will be about !{tags:0}

# !{date:property} - accessing property of date
This article was originally published on !{date:published}
Thoroughly updated through this website on !{date:updated}

There should only be one <h1> heading per page, and it’s usually declared in the front matter as title, which is why headings in the content starts at 2 ## (equivalent to <h2>) with the lowest one being 4 #### (equivalent to <h4>) and should conform with the rules of markdownlint, with some essential ones to follow are

MD001: Heading levels should only increment by one level at a time
MD003: Heading style; only ATX style
MD018: No space after hash on atx style heading
MD023: Headings must start at the beginning of the line
MD024: Multiple headings with the same content; siblings only
MD042: No empty links

Generated ids can be specified from the text by wrapping them in $(...) as the delimiter. The text inside will be converted to kebab-case and will be used as the id. If no delimiter is detected, the whole text will be used.

If you’re using VSCode, you can install the markdownlint extension to help you catch these lint errors / warnings and write better markdown. These rules can be configured, see the .jsonc template and .yaml template with an example here.

Code Blocks

Code blocks are fenced with 3 backticks and can optionally be assigned a language for syntax highlighting. The language must be a valid shiki supported language and is case-insensitive.

	
		
			
			
		
	

	```language
// code
```

Additional information can be added to the code block through data attributes, accessible via data-[key]="[value]". The dataset can be specified from any line within the code block using #$ key: value syntax, and it will be omitted from the output. The key-value pair should roughly conform to the data-* rules, meaning key can only contain alphanumeric characters and hyphens, while value can be any string that fits in the data attribute value.

There are some special keys that will be used to modify the code block itself, and they are

#$ file: string | add a filename to the code block that will be shown above the output
#$ line-start: number | define the starting line number of the code block

Module / Core

Marqua provides a lightweight core module with minimal features and dependencies that does not rely on platform-specific modules so that it could be used anywhere safely.

parse

Where the parsing happens, it accepts a source string and returns a { content, metadata } structure. This function is mainly used to separate the front matter from the content.

	
		
			
			
		
	

	export function parse(source: string): {
	content: string;
	metadata: Record<string, any> & {
		readonly estimate: number;
		readonly table: MarquaTable[];
	};
};

If you need to read from a file or folder, use the compile and traverse functions from the FileSystem module.

construct

Where the metadata or front matter index gets constructed, it is used in the parse function.

	
		
			
			
		
	

	type Primitives = null | boolean | string;
type ValueIndex = Primitives | Primitives[];
type FrontMatter = { [key: string]: ValueIndex | FrontMatter };

export function construct(raw: string): ValueIndex | FrontMatter;

Module / Artisan

transform

This isn’t usually necessary, but in case you want to handle the markdown parsing and rendering by yourself, here’s how you can tap into the transform function provided by the module.

	
		
			
			
		
	

	export interface Dataset {
	lang?: string;
	file?: string;
	[data: string]: string | undefined;
}

export function transform(source: string, dataset: Dataset): string;

A simple example would be passing a raw source code as a string.

	
		
			
			
		
	

	import { transform } from 'marqua/artisan';

const source = `
interface User {
	id: number;
	name: string;
}

const user: User = {
	id: 0,
	name: 'User'
}
`;

transform(source, { lang: 'typescript' });

Another one would be to use as a highlighter function.

	
		
			
			
		
	

	import MarkdownIt from 'markdown-it';
import { transform } from 'marqua/artisan';

// passing as a 'markdown-it' options
const marker = MarkdownIt({
	highlight: (source, lang) => transform(source, { lang });
});

marker

The artisan module also exposes the marker import that is a markdown-it object.

	
		
			
			
		
	

	import { marker } from 'marqua/artisan';
import plugin from 'markdown-it-plugin'; // some markdown-it plugin
marker.use(plugin); // add this before calling 'compile' or 'traverse'

Importing marker to extend with plugins is optional, it is usually used to enable you to write LaTeX in your markdown for example, which is useful for math typesetting and writing abstract symbols using TeX functions. Here’s a working example with a plugin that uses KaTeX.

	
		
			
			
		
	

	import { marker } from 'marqua/artisan';
import { compile } from 'marqua/fs';
import TexMath from 'markdown-it-texmath';
import KaTeX from 'katex';

marker.use(TexMath, {
	engine: KaTeX,
	delimiters: 'dollars',
});

const data = compile(/* source path */);

Module / Browser

hydrate

This is the browser module to hydrate and give interactivity to your HTML.

	
		
			
			
		
	

	import type { ActionReturn } from 'svelte/action';

export function hydrate(node: HTMLElement, key: any): ActionReturn;

The hydrate function can be used to make the rendered code blocks from your markdown interactive, some of which are

toggle code line numbers
copy block to clipboard

Usage using SvelteKit would simply be

	
		
			
			
		
	

	<script>
	import { hydrate } from 'marqua/browser';
	import { navigating } from '$app/stores';
</script>

<main use:hydrate={$navigating}>
	<!-- content here -->
</main>

Passing in the navigating store into the key parameter is used to trigger the update inside hydrate function and re-hydrate the DOM when the page changes but is not remounted.

Module / FileSystem

Marqua provides a couple of functions coupled with the FileSystem module to compile or traverse a directory, given an entry point.

Using a folder structure shown below as a reference for the next examples, the usage will be as follows

	
		
			
			
		
	

	content
    ├── posts
    │   ├── draft.my-amazing-two-part-series-part-1.md
    │   ├── draft.my-amazing-two-part-series-part-2.md
    │   ├── 2021-04-01.my-first-post.md
    │   └── 2021-04-13.marqua-is-the-best.md
    └── reviews
        ├── game
        │   └── doki-doki-literature-club.md
        ├── book
        │   ├── amazing-book-one.md
        │   └── manga-is-literature.md
        └── movie
            ├── spirited-away.md
            └── your-name.md

compile

	
		
			
			
		
	

	interface HydrateChunk {
	breadcrumb: string[];
	buffer: Buffer;
	parse: typeof parse;
}

export function compile(
	entry: string,
	hydrate?: (chunk: HydrateChunk) => undefined | Output,
): undefined | Output;

The first argument of compile is the source entry point.

traverse

	
		
			
			
		
	

	export function traverse(
	options: {
		entry: string;
		compile?(path: string): boolean;
		depth?: number;
	},
	hydrate?: (chunk: HydrateChunk) => undefined | Output,
	transform?: (items: Output[]) => Transformed,
): Transformed;

The first argument of traverse is its typeof options and the second argument is an optional hydrate callback function. The third argument is an optional transform callback function.

The compile property of the options object is an optional function that takes the full path of a file from the entry point and returns a boolean. If the function returns true, the file will be processed by the compile function, else it will be passed over to the hydrate function if it exists.

An example usage from the hypothetical content folder structure above should look like

	
		
			
			
		
	

	import { compile, traverse } from 'marqua/fs';

/* compile - parse a single source file */
const body = compile(
	'content/posts/2021-04-01.my-first-post.md',
	({ breadcrumb: [filename], buffer, parse }) => {
		const [date, slug] = filename.split('.');
		const { content, metadata } = parse(buffer.toString('utf-8'));
		return { ...metadata, slug, date, content };
	},
); // {'posts/2021-04-01.my-first-post.md'}

/* traverse - scans a directory for sources */
const data = traverse({ entry: 'content/posts' }, ({ breadcrumb: [filename], buffer, parse }) => {
	if (filename.startsWith('draft')) return;
	const [date, slug] = filename.split('.');
	const { content, metadata } = parse(buffer.toString('utf-8'));
	return { ...metadata, slug, date, content };
}); // [{'posts/3'}, {'posts/4'}]

/* traverse - nested directories infinite recursive traversal */
const data = traverse(
	{ entry: 'content/reviews', depth: -1 },
	({ breadcrumb: [slug, category], buffer, parse }) => {
		const { content, metadata } = parse(buffer.toString('utf-8'));
		return { ...metadata, slug, category, content };
	},
); // [{'game/0'}, {'book/0'}, {'book/1'}, {'movie/0'}, {'movie/1'}]

Module / Transform

This module provides a set of transformer functions for the traverse.transform parameter. These functions can be used in conjunction with each other, by utilizing the pipe function provided from the 'mauss' package and re-exported by this module, you can do the following

	
		
			
			
		
	

	import { traverse } from 'marqua/fs';
import { pipe } from 'marqua/transform';

traverse({ entry: 'content' }, () => {}, pipe(/* ... */));

chain

The chain transformer is used to add a flank property to each items and attaches the previous (idx - 1) and the item after (idx + 1) as flank: { back, next }, be sure to sort it the way you intend it to be before running this transformer.

	
		
			
			
		
	

	export function chain<T extends { slug?: string; title?: any }>(options: {
	base?: string;
	breakpoint?: (next: T) => boolean;
	sort?: (x: T, y: T) => number;
}): (items: T[]) => Array<T & Attachment>;

A base string can be passed as a prefix in the slug property of each items.

A breakpoint function can be passed to stop the chain on a certain condition.

	
		
			
			
		
	

	traverse(
	{ entry: 'content' },
	({}) => {},
	chain({
		breakpoint(item) {
			return; // ...
		},
	}),
);

A sort function can be passed to sort the items before chaining them.