Slurp

Slurps webpages, cleans off all the crud, and saves them to Obsidian as nice, tidy Markdown files. Think Pocket, but better.

Features

• Slurp pages into Obsidian using the command palette, bookmarklets, or browser extensions (soon™️).
• Customize frontmatter properties with formatting options or by adding your own.
• Enrich slurped pages with frontmatter automatically using metadata sources like OpenGraph, including:
  • Tags - Create customizable tags from keywords present on the page.
  • Excerpt - Often used for subtitles, excerpts, descriptions, and abstracts.
  • Byline - Name of the primary author or the first author detected.
  • Site name - Website or publication name. Useful for blogs and news sites.
  • Published and Modified dates
  • Permalink
  • Page type - The page type, usually "article", "page", or "post".
  • Twitter handle
  • Onion mirrors
  • and more...
• Works on desktop and mobile devices.

Usage

Detailed usage information can be found in the documentation.

Create Note from URL

1. Ctrl+P or Cmd+P to open the command palette
2. Select Slurp: Create note from URL
3. Paste the URL and hit Enter or tap the Slurp button

Using Bookmarklets or the Browser Extension (Soon™️)

Slurp exposes a custom URI which can be used for one-click saves.

Bookmarklets are a simple option for those who prefer them. Simply create a new bookmark with the following URL set:

javascript:(() => document.location.href=`obsidian://slurp?url=${document.URL}`)();

Browser extensions are also coming soon for Firefox and Chrome-compatible browsers. If you want to get started right away, they are currently in beta and can be manually installed. See https://github.com/inhumantsar/slurp-extension for details.

Settings

FrontMatter Properties

Note properties are used by Obsidian to add metadata to notes. Supported data types include checkboxes (true and false values), dates and datetimes, lists, numbers, and good ol' plaintext.

By default, Slurp will try to find relevant metadata and add it to new notes. The plugin settings screen offers a few ways to adjust how this metadata is handled and presented:

• Edit property keys to define what name to use for the metadata.
• Tap the up and down arrows to customize the order in which properties appear in notes.
• Use the Show empty properties toggle to get Slurp to add properties even if there is no data to populate them with.
• Selectively enable/disable individual properties.
• Customize how properties are formatted.
• Add custom properties.

For more information, check out the documentation.

Roadmap

Toward v1

• Add settings to customize and selectively disable properties.
• Improve documentation and project structure.
• Add setting for default save location.
• IN PROGRESS Browser extension for one-click slurps.
• Offer tag parsing, tag prefix, and save location options at slurp-time.
• Import Pocket saves, bookmarks, and more automagically
• Support for multiple authors in the byline field.
• Use a bit of custom parsing logic for popular sites to capture better data and tidy up results:
  • arXiv: Authors, topics, arXiv IDs, dates, and cleaner formatting. Stretch goal: Grab the paper PDF and any code links as well.
  • Medium: Clean up the author information captured, particularly the links which get spread across multiple lines currently.

Beyond v1

• Ensure video and other embeds are captured reliably
• Integrate with an LLM to provide summaries and tag recommendations
• Make sure Slurp plays nicely with other plugins, eg Dataview
• Save PDF and/or HTML versions of the page with the Markdown versions
• More custom parsing logic
  • HackerNews: Map discussion threads to blockquote levels, capture both the HN URL and the article URL, use submitter name in the byline, ensure dates are reliably captured. Stretch goal: Scores, capture article along with the discussion.
  • Reddit: Literally any actual content, plus everything mentioned for HN.

Beta Testing

If you would like to help test new features before they are officially released:

1. Install BRAT from the Community Plugins directory
2. Open the command palette and run the command BRAT: Add a beta plugin for testing.

• Do not use a frozen version! I don't tag pre-releases.

3. Enter this repository's URL, ie: https://github.com/inhumantsar/slurp.

BRAT will regularly look for updates and install them. This can be configured/disabled in the BRAT settings menu.

Development Environment

Slurp does a couple things differently from the standard Obsidian plugin development setup:

• Svelte 5 is used to build the property settings component
• The Typescript libraries have been updated to v5.4

If you are a plugin developer already, using a separate environment for Slurp is recommended.

Code Style

The Zen of Python is a great styleguide for any language.

• Beautiful is better than ugly.
• Explicit is better than implicit.
• Simple is better than complex.
• Complex is better than complicated.
• Flat is better than nested.
• Sparse is better than dense.
• Readability counts.
• Special cases aren't special enough to break the rules.
• Although practicality beats purity.
• Errors should never pass silently.
• Unless explicitly silenced.
• In the face of ambiguity, refuse the temptation to guess.
• There should be one-- and preferably only one --obvious way to do it.
• Although that way may not be obvious at first unless you're Dutch.
• Now is better than never.
• Although never is often better than right now.
• If the implementation is hard to explain, it's a bad idea.
• If the implementation is easy to explain, it may be a good idea.
• Namespaces are one honking great idea -- let's do more of those!

When it comes to Typescript specifically, I try to follow the guidelines below. Take these with a grain of salt though. I'm still new to Typescript though and I don't have a ton of professional experience with Javascript generally. If any of these are superdumb, please let me know!

• Don't use @ts-ignore unless it's absolutely necessary.
• Add interfaces whenever complex data types are passed between functions.
• Prefer interfaces over types.
• Prefix interface names with I and type names with T.
• Compact structures and anonymous functions are preferred, eg: map(...) > for (...) {}.
• Descriptive type, function, and variable names are preferred
• 1-3 character names are fine in small scopes, eg: (k, v) => {...} and for (let i in somevar).
• KISS: Any function longer than 10-15 lines or with more than 1 or 2 levels of indentation should probably be broken down.
• Lines over 140 characters long should be broken up.

Also:

• Please don't use Prettier.

direnv

There is a direnv config which can be used to quickly configure a completely isolated local environment. Setting it up requires a few extra steps though.

1. Install the Nix package manager: sh <(curl -L https://nixos.org/nix/install) --no-daemon
2. Ensure flakes and nix-command are enabled, eg: mkdir -p ~/.local/nix && echo "experimental-features = nix-command flakes" >> nix.conf
3. Install direnv, adjusting or removing bin_path as needed: curl -sfL https://direnv.net/install.sh | bin_path=~/.local/bin bash
4. direnv will instruct you to add a line to your .bashrc, once that's done, run direnv allow.

Building

npm install     # not required if using direnv
npm run dev     # enable hot-rebuilds of main.js

Versioning

The usual semantic versioning applies.

manifest-beta.json provides the dev channel specifications for BRAT.

Test Vault

test-resources/vault is an Obisidian vault that can be used for testing. As a side-benefit, it's a place to keep development notes.

There is a symlink in the vault's plugins directory which uses a relative path to reference the repository root. This may or may not work for you after cloning. Remove and recreate it if Obsidian doesn't see the plugin properly.

NOTE: The plugin won't work (and may not even be recognized) if you haven't built the project yet!

Hot Reload

Hot Reload is a commonly used plugin for Obsidian plugin development. It will watch for modified plugins and automatically reload it within a running Obsidian instance. It's included in the test vault as a submodule, so you will need to update it on first clone:

git submodule update

Testing

URI Handler

On Linux:

xdg-open "obsidian://slurp?url=https://..."

Credits

• Mozilla's Readability powers the underlying conversion.

License

MIT