Mailing subtrees with Attachments

This is pretty exciting; I’ve figured out how to quickly send org-mode subtrees as MIME-encoded emails. That means that, essentially, I can write in org, as plain text, and very quickly export to HTML, add attachments, and send. The exciting part about this for me is that it should streamline my communications with students, while also letting me stay in Org and keep my records in order. Let’s walk through the process.

Use-case

For the moment, I still use Thunderbird as my primary MUA. It’s pretty easy to use, minimal configuration compared to all things Emacs, and if something goes wrong with it I don’t have to quit Emacs (!), just Thunderbird.

In some cases, though, Thunderbird makes for an awkward workflow. That’s certainly the case for grading, whih has many, poorly-integrated elements. To mark an assignment I need to:

  • log in to Blackboard (in Firefox; I wonder if I could do that in Emacs?)
  • download the set of student papers (one at a time, 2 clicks per paper) to Downlads
  • move papers to a directory (usually ~/COURSENAME/Grading/ASSIGNMENTNAME )
  • Read papers in LigreOffice, comment inline
  • record mark in Libreoffice Calc spreadsheet
  • email paper back to student with comments
  • upload marks back into Blackboard
  • find a place to archive the student paper in case I need it later, e.g. for a contested grade.

The while process basically sucks. I spend maybe 20% of my time fussing with paths and mouse clicks and email addresses. So I am experimenting with moving as much of this process into Emacs. So far, I don’t think there’s any way at all to bulk-download the papers – that sucks, but I can live with it I guess (I have to!). So I start the optimization at the point where I have all my papers ready to go in a subdir.

Org-mime

Org-mime is the library that allows org buffers and other elements to be quickly converted to HTML and prepared for multi-part messaging. Load it and set it up (see http://orgmode.org/worg/org-contrib/org-mime.html):

;; enable HTML email from org
(require 'org-mime)
;; setup org-mime for wanderlust
;; (setq org-mime-library 'semi)
;; or for gnus/message-mode
(setq org-mime-library 'mml)

;; easy access to htmlize in message-mode
(add-hook 'message-mode-hook
          (lambda ()
            (local-set-key "\C-c\M-o" 'org-mime-htmlize)))

;; uncomment this to use the org-mome native functions for htmlizing.
;; (add-hook 'org-mode-hook
;;           (lambda ()
;;             (local-set-key "\C-c\M-o" 'org-mime-org-buffer-htmlize)))

;; uncomment to displyay src blocks with a dark background
;; (add-hook 'org-mime-html-hook
;;           (lambda ()
;;             (org-mime-change-element-style
;;              "pre" (format "color: %s; background-color: %s; padding: 0.5em;"
;;                            "#E6E1DC" "#232323"))))

;; pretty blockquotes
(add-hook 'org-mime-html-hook
          (lambda ()
            (org-mime-change-element-style

             "blockquote" "border-left: 2px solid gray; padding-left: 4px;")))

Fix htmlization

Upstream org-mime-htmlize unfortunately can’t be called uninteractively (bummer!), so we have to rewrite it to make programmatic calls work properly. I found the solution Emacs Stackexchange.

(defun org-mime-htmlize (&optional arg)
"Export a portion of an email body composed using `mml-mode' to
html using `org-mode'.  If called with an active region only
export that region, otherwise export the entire body."
  (interactive "P")
  (require 'ox-org)
  (require 'ox-html)
  (let* ((region-p (org-region-active-p))
         (html-start (or (and region-p (region-beginning))
                         (save-excursion
                           (goto-char (point-min))
                           (search-forward mail-header-separator)
                           (+ (point) 1))))
         (html-end (or (and region-p (region-end))
                       ;; TODO: should catch signature...
                       (point-max)))
         (raw-body (concat org-mime-default-header
                           (buffer-substring html-start html-end)))
         (tmp-file (make-temp-name (expand-file-name
                                    "mail" temporary-file-directory)))
         (body (org-export-string-as raw-body 'org t))
         ;; because we probably don't want to export a huge style file
         (org-export-htmlize-output-type 'inline-css)
         ;; makes the replies with ">"s look nicer
         (org-export-preserve-breaks org-mime-preserve-breaks)
         ;; dvipng for inline latex because MathJax doesn't work in mail
         (org-html-with-latex 'dvipng)
         ;; to hold attachments for inline html images
         (html-and-images
          (org-mime-replace-images
           (org-export-string-as raw-body 'html t) tmp-file))
         (html-images (unless arg (cdr html-and-images)))
         (html (org-mime-apply-html-hook
                (if arg
                    (format org-mime-fixedwith-wrap body)
                  (car html-and-images)))))
    (delete-region html-start html-end)
    (save-excursion
      (goto-char html-start)
      (insert (org-mime-multipart
               body html (mapconcat 'identity html-images "\n"))))))

Actually perform the export!

These functions are crude helpers that gather extra information about the org subtree, of which org-mime is unaware.

  • mwp-org-get-parent-headline traverses the tree to the ancestor headline, because that’s what I want to set the subject to.
  • mwp-org-attachment-list is stolen directly from the Gnorb package, which looks cool, awesome ,and kinda complex; it just iterates through a subtree’s attachments and grabs URLs.
  • mwp-send-subtree-with-attachments performs the export and is bound to C-c M-o

So, if I want to mail a subtree, I just C-c M-o and I’m almost done – the html mail is ready to go, and all org attachments are also attached to the email.

Note there are some real weaknesses here: mwp-org-get-parent-headline actually gets the top-level ancestor – which only happens to be what I want right now. Better would be to use org-element to locate the parent (and other headline attributes) directly, but I’m not sure how to do that.

Similarly, the initial greeting is generated from the current headline value – so this only works because I name my subtrees after the addressee (which I only do because of my use case).

(defun mwp-org-get-parent-headline ()
  "Acquire the parent headline & return."
  (save-excursion
    (org-mark-subtree)
    (re-search-backward  "^\\* ")
    (nth 4 (org-heading-components))))

(defun mwp-send-subtree-with-attachments ()
  "org-mime-subtree and HTMLize"
  (interactive)
  (org-mark-subtree)
  (let ((attachments (mwp-org-attachment-list))
        (subject  (mwp-org-get-parent-headline)))
    (insert "Hello " (nth 4 org-heading-components) ",\n")
    (org-mime-subtree)
    (insert "\nBest,\nMP.\n")
    (message "subject is" )
    (message subject)
    ;;(message-to)
    (org-mime-htmlize)
    ;; this comes from gnorb
    ;; I will reintroduce it if I want to reinstate questions.
    ;; (map-y-or-n-p
    ;;  ;; (lambda (a) (format "Attach %s to outgoing message? "
    ;;  ;;                    (file-name-nondirectory a)))
    ;; (lambda (a)
    ;;   (mml-attach-file a (mm-default-file-encoding a)
    ;;                    nil "attachment"))
    ;; attachments
    ;; '("file" "files" "attach"))
    ;; (message "Attachments: %s" attachments)
    (dolist (a attachments) (message "Attachment: %s" a) (mml-attach-file a (mm-default-file-encoding a) nil "attachment"))
    (message-goto-to)
    ))

;; add a keybinding for org-mode
(add-hook 'org-mode-hook
          (lambda ()
            (local-set-key "\C-c\M-o" 'mwp-send-subtree-with-attachments)))

;; stolen from gnorb; finds attachments in subtree
(defun mwp-org-attachment-list (&optional id)
  "Get a list of files (absolute filenames) attached to the
current heading, or the heading indicated by optional argument ID."
  (when (featurep 'org-attach)
    (let* ((attach-dir (save-excursion
                         (when id
                           (org-id-goto id))
                         (org-attach-dir t)))
           (files
            (mapcar
             (lambda (f)
               (expand-file-name f attach-dir))
             (org-attach-file-list attach-dir))))
      files)))

Contacts

That’s a good start, but there are still some steps to make this truly convenient. For instance, I certainly don’t want to type in students’ email addresses by hand. So I imported my contacts from thunderbird to org-contacts. This was a pain – the process was Thunderbird → Gmail (via gsync plugin) → vcard (via gmail export) → org-contacts (via Titus’s python importer). I wish there was a CSV importer for org-contacts; probably this would be easy to write but I’m so slooooowwww at coding. My org contacts live in GTD/Contacts.org, which is set in Customize, and org reads them on startup with this line

(require 'org-contacts)

With this single line, org-contacts now provides TAB completion in message-mode to headers. It’s very fast, so feels more convenient than Thunderbird.

Making it better
I wish I could get org-contacts to provide tab completion in my subtrees (see below). I would need to access the completion function directly and somehow set the binding for TAB to that completion function.

Adding Attachments with Drag & Drop

After I make inline comments, I fill out a grading template and attach the paper to the resultant subtree (C-c C-a a PATH). This is OK, but sometimes it would nice to be able to drag and drom the files, so I am working on these functions.

Even better
an even better solution be to add the attachments programmatically. The studnet papes follow a strict naming convention, so I should be able to crawl the directory and find the most recent paper with the student’s name in it… I’m worried it wil lbe too error prone though.

Anyway: unfortunately the following code doesn’t work right, so don’t just cut and paste this code!). I *ought to be able to bind the drag and drop action to a function – even several functions – and, if conditions are right, attach the dragged file to the current org header. John Kitchin describes this method here. But I do the following instead, which is also broken right now:

Start by loading org-download, which downloads dragged images as attachments and inserts a link. (yay). THen a modification which fixes handling of file links allowing me to drag-n-drop files links onto org as attachments. Unfortunately, I can’t get org-attach to process the URI’s properly. Darn it.

(require 'org-download)
(require 'org-attach)
;; extending the dnd functionality
;; but doesn't actually work... 
(defun mwp-org-file-link-dnd (uri action)
    "When in `org-mode' and URI points to local file, 
  add as attachment and also add a link. Otherwise, 
  pass URI and Action back to dnd dispatch"
    (let ((img-regexp "\\(png$\\|jp[e]?g$\\)")
          (newuri (replace-regexp-in-string "file:///" "/" uri)))
      (cond ((eq major-mode 'org-mode)
             (message "Hi! newuri: %s " (file-relative-name newuri))
             (cond ((string-match img-regexp newuri)
                    (insert "#+ATTR_ORG: :width 300\n")
                    (insert (concat  "#+CAPTION: " (read-input "Caption: ") "\n"))
                    (insert (format "[[%s]]" uri))
                    (org-display-inline-images t t))
                   (t 
                    (org-attach-new newuri)
                    (insert (format "[[%s]]" uri))))
             )
            (t
             (let ((dnd-protocol-alist
                    (rassq-delete-all
                     'mwp-org-file-link-dnd
                     (copy-alist dnd-protocol-alist))))
               (dnd-handle-one-url nil action uri)))
            )))

  ;; add a new function that DOESN'T open the attachment!
(defun org-attach-new-dont-open (file)
    "Create a new attachment FILE for the current task.
  The attachment is created as an Emacs buffer."
    (interactive "sCreate attachment named: ")
    (when (and org-attach-file-list-property (not org-attach-inherited))
      (org-entry-add-to-multivalued-property
       (point) org-attach-file-list-property file))
    )

(defun mwp-org-file-link-enable ()
    "Enable file drag and drop attachments."
    (unless (eq (cdr (assoc "^\\(file\\)://" dnd-protocol-alist))
                'mwp-org-file-link-dnd)
      (setq dnd-protocol-alist
            `(("^\\(file\\)://" . mwp-org-file-link-dnd) ,@dnd-protocol-alist))))

(defun mwp-org-file-link-disable ()
  "Enable file drag and drop attachments."
  (if (eq (cdr (assoc "^\\(file\\)://" dnd-protocol-alist))
              'mwp-org-file-link-dnd)
      (rassq-delete-all
       'mwp-org-file-link-dnd
       dnd-protocol-alist)

    ))

(mwp-org-file-link-enable)

Exporting org-files to a temporary location

I have a private journal, which lives in an encrypted file in a Dropbox-backed-up directory. I use html export to examine the contents sometimes – there are some big old tables that are hard to read in org-mode – but I don’t want the html file to end up in Dropbox.

So I just copied the definition of org-export-html-as-html and made trivial modifications. There’s probably a better way to do this.

;; export html to tmp dir
(defun mwp-org-html-to-tmp
    (&optional async subtreep visible-only body-only ext-plist)
  "Export current buffer to a HTML file in the tmp directory.

If narrowing is active in the current buffer, only export its
narrowed part.

If a region is active, export that region.

A non-nil optional argument ASYNC means the process should happen
asynchronously.  The resulting file should be accessible through
the `org-export-stack' interface.

When optional argument SUBTREEP is non-nil, export the sub-tree
at point, extracting information from the headline properties
first.

When optional argument VISIBLE-ONLY is non-nil, don't export
contents of hidden elements.

When optional argument BODY-ONLY is non-nil, only write code
between \"<body>\" and \"</body>\" tags.


EXT-PLIST, when provided, is a property list with external
parameters overriding Org default settings, but still inferior to
file-local settings.

Return output file's name."
  (interactive)
  (let* ((extension (concat "." (or (plist-get ext-plist :html-extension)
                                    org-html-extension
                                    "html")))
;; this is the code I've changed from the original function. 
         (file (org-export-output-file-name extension subtreep "/home/matt/tmp/"))

         (org-export-coding-system org-html-coding-system))
    (org-export-to-file 'html file
      async subtreep visible-only body-only ext-plist)
    (org-open-file file)))

(org-defkey org-mode-map
            (kbd "C-c 0") 'mwp-org-html-to-tmp)

Creating and Publishing Presentations with Org-reveal

For several years, I’ve been using Org-mode to compose slides for my lectures. This method is great, because I get to work in plain-text and focus on the content of my lectures rather than animations; but it’s meant that when I want to share my presentations with others, there’s a certain amount of work involved as I move from a local copy on my computer to a web-based version. (This has largely been an issue because I sometimes compose my lectures sitting in a café with lousy Internet, and I sometimes give my lectures in a horrible classroom at U of T with terrible Internet reception.) I’ve now largely solved this problem, though there is hopefully an improvement coming down the pipe which will make it even easier.

Org-mode has the capacity to export to a number of slide-like formats, including the LaTeX-based Beamer format, which also makes good PDF presentations, a couple of Emacs-based presentation tools, and a number of HTML5 formats. Since I teach about the web all the time, the HTML5 formats have always been the most appealing to me.

Org-Reveal Setup

I have used and still very much like deck.js (exporter here), but have recently switched to org-reveal, which I really like a lot. It’s not part of the official org distribution, so installation and setup are a little more involved, but not difficult. I just cloned the org-reveal and reveal.js repositories:

cd ~/src
git clone https://github.com/hakimel/reveal.js.git
git clone https://github.com/yjwen/org-reveal.git

and put this in my emacs-init.el:

;; org-reveal
(add-to-list 'load-path "~/src/org-reveal")
(require 'ox-reveal)
;; set local root
(setq org-reveal-root "file:///home/matt/src/reveal.js")

That’s all that’s needed to get export working! I find it’s really fast to prepare lectures.

Publishing

That’s great for giving lectures, and is all I really need at 9:55 when I’m trying to type my lecture and walk to class at the same time. But after lecture I want to put my slides somewhere my students can see them. Even if I wanted to, it would be impossible for me to post to Blackboard, which turns these files into garbage. What I want to do is publish them to the web; but I need to make sure that all the JS and CSS links are pointing to the web-based libraries and not my local copies, which of course no one but me can see. To do this I had to make one small change to org-reveal.el, which I have submitted as a pull request. This creates a new variable, org-reveal-extra-css, which I can refer to in my own functions.

Then I use org-mode’s fantastic built-in Publishing functions to push my slides to a public website. Publishing allows you to perform an export on many files, and customize the output in powerful ways that are mostly beyond me, actually. Still, I have a setup that I like a lot.

First, my org-publish-project-alist, which defines the publishing targets. Note especially the top part, which defines “meta-projects”: for instance, I can publish all the slides and source files for all my classes with one command, M-x org-publish-projects [RET] courses.

(setq org-publish-project-alist
      '(
        ("courses"
         :components ("dh" "rlg231"))
        ("rlg231"
         :components ("rlg231-lecture-slides" "rlg231-lecture-source"))
        ("dh"
         :components ("digital-history-lecture-slides" "digital-history-lecture-source"))

        ("rlg231-lecture-slides"
         :base-directory "~/RLG231/Lectures/"
         :base-extension "org"
         :publishing-directory "/ssh:matt@shimano:/var/www/sandbox/RLG231/Lectures/Slides"
         :recursive t
         :publishing-function mwp-org-reveal-publish-to-html
         :preparation-function nil 
         :completion-function nil
         :headline-levels 4             ; Just the default for this project.
         :exclude "LectureOutlines.org"
         :exclude-tags note noexport
         :auto-preamble t)

        ("rlg231-lecture-source"
         :base-directory "~/RLG231/Lectures/"
         :base-extension "org"
         :publishing-directory "/ssh:matt@shimano:/var/www/sandbox/RLG231/Lectures/Source"
         :recursive t
         :publishing-function org-org-publish-to-org
         :preparation-function nil
         :completion-function nil
         :headline-levels 4             ; Just the default for this project.
         :exclude "LecturePlans.org"
         ;; :exclude "LectureOutlines.org"
         :exclude-tags note noexport
         :auto-preamble t)

        ("digital-history-lecture-source"
         :base-directory "~/DH/Lectures"
         :base-extension "org"
         :publishing-directory "/ssh:matt@shimano:/var/www/sandbox/DigitalHistory/Lectures/Source"
         :recursive t
         :publishing-function org-org-publish-to-org
         :preparation-function 
         :completion-function 
         :headline-levels 4             ; Just the default for this project.
         ;; :exclude "LecturePlans.org"
         :exclude "LectureOutlines.org"
         :exclude-tags note noexport
         :auto-preamble t)

        ("digital-history-lecture-slides"
         :base-directory "~/DH/Lectures"
         :base-extension "org"
         :publishing-directory "/ssh:matt@shimano:/var/www/sandbox/DigitalHistory/Lectures/Slides"
         :recursive t
         :publishing-function mwp-org-reveal-publish-to-html
         :preparation-function 
         :completion-function 
         :headline-levels 4             ; Just the default for this project.
         ;; :exclude "LecturePlans.org"
         :exclude "LectureOutlines.org"
         :exclude-tags note noexport
         :auto-preamble t)

        ;; ("newone-lecture-slides"
        ;;  :base-directory "~/NewOne/Lectures/"
        ;;  :base-extension "org"
        ;;  :publishing-directory "/ssh:matt@shimano:/var/www/sandbox/NewOne/Lectures"
        ;;  :recursive t
        ;;  :publishing-function org-deck-publish-to-html
        ;;  :headline-levels 4             ; Just the default for this project.
        ;;  :exclude-tags note noexport
        ;;  :auto-preamble t)

        ;; ("newone-lecture-notes"
        ;;  :base-directory "~/NewOne/Lectures/"
        ;;  :base-extension "org"
        ;;  :publishing-directory "/ssh:matt@shimano:/var/www/sandbox/NewOne/Lectures-with-notes"
        ;;  :recursive t
        ;;  :publishing-function org-html-publish-to-html
        ;;  :headline-levels 4             ; Just the default for this project.
        ;;  :exclude-tags noexport
        ;;  :auto-preamble t)

        ;;  ("newone-images"
        ;;        :base-directory "~/NewOne/Images/"
        ;;        :base-extension "jpg\\|gif\\|png"
        ;;        :publishing-directory "/ssh:matt@shimano:/var/www/sandbox/NewOne/Images"
        ;;        :publishing-function org-publish-attachment)

        ;;  ("newone" :components ("newone-lecture-slides" "newone-lecture-notes" "newone-images") )

        ;;  ("presentations"
        ;;   :base-directory "~/Dropbox/Work/Talks/"
        ;;   :base-extension "org"
        ;;   :publishing-directory "/ssh:matt@shimano:/var/www/sandbox/Presentations"
        ;;   :headline-levels 4 ; just the default for this project
        ;;   :exclude-tags noexport
        ;;   :auto-preamble t
        ;;   :publishing-function mwp-org-deck-publish-to-html
        ;;   ;; :completion-function mwp-update-published-paths
        ;;   )

      ))

Notice the publishing function, which is set to mwp-org-deck-publish-to-html. This is a simple function that resets the base url and extra-css values to web-based ones before publication, so that the presentations work when online. Notice I’ve also reset the deck.js base url, in case I ever decide to change back to deck.

(defun mwp-org-reveal-publish-to-html (plist filename pub-dir)
  "Publish an org file to reveal.js HTML Presentation.
FILENAME is the filename of the Org file to be published.  PLIST
is the property list for the given project.  PUB-DIR is the
publishing directory. Returns output file name."
  (let ((org-deck-base-url "http://sandbox.hackinghistory.ca/Tools/deck.js/")
        (org-reveal-root "http://sandbox.hackinghistory.ca/Tools/reveal.js/")
        (org-reveal-extra-css "http://sandbox.hackinghistory.ca/Tools/reveal.js/css/local.css"))

    (org-publish-org-to 'reveal filename ".html" plist pub-dir))
  )

And that’s it, magic!

Still to do

I like this a lot, but there are a couple of pieces I’d still like to implement.

Fix all local file URL’s
I’d like to write a function to take a final pass through all the links and change file:/// links to HTML relative links. That will take some work though.
Export as standalone
There is work underway to allow presentations to be generated as stand-alone files that can be, e.g, sent by email. I like this idea a lot. See this Github issue.
Standardize notes, fragments
Every time I switch from one presentation framework to another, I have to learn a whole different syntax for things like fragments (bits of content that don’t appear on the slide immediately, but are instead stepped through) and speaker notes (that don’t appear on the slide that your viewers see, but are only visible to you in some kind of preview mode). It would be great if the various slide modes could work towards a common syntax for these things. If I have time, energy, and skills, I would like to help develop this a little.

See my slides

If you want to see some examples of the end product, here is a link to my Digital History lecture archive (still being built!). Many of my course materials are also online at Github.