It’s been a while since I maintained this blog actively, but some of these old posts are still referenced by Emacs users, so I have migrated them over from my old Wordpress site. The migration process produced a lot of errors and I haven’t had time to fix them all up yet, but I’m working my way through as best I can.

Yes, I know, It’s been a while

This is an ancient WordPress blog which I at one point hoped would be a more comprehensive representation of my activity online. I’m increasingly frustrated with WordPress, though, and while there are excellent tools to work in the framework from e.g. Emacs, I’d still rather maintain a simple plain-text static site. However… self-promotion never seems to bubble to the top of my to-do list. This notice is just to say: SOME DAY you will see a Hugo or Jekyll site here, and when that day comes ,perhaps I’ll start blogging actively again. Thanks for dropping by!

Deploying Course Websites with git hooks.

Table of Contents Repository setup Workflow Now that I’ve moved my course websites to Hugo via ox-hugo, life is definitely easier than when I used WordPress, and my teaching is unquestionably crawling a little bit closer to actual reproducibility. It’s been non-trivial to figure out a good deployment process, though. My course materials live in a slightly broader context than my websites: I have my readings, my private notes, my grades, and a few other documents like teaching contracts, etc., in the main course directory. Some but not all of these materials can be committed to Git, so it makes intuitive sense to me to have a course repository which exists independently of the website. If I didn’t use ox-hugo to manage the org-to-hugo process – if, say, I used the somewhat more limited googrgeous support which is built into Hugo – I might try to build the website directly from the course materials. Instead, I’ve come up with a somewhat more convoluted process. It’s by no means perfect, and there are probably better solutions , but for now this process is working pretty well for me. Documenting here in case it’s useful to anyone else, and also in case I forget what I’ve done! For this post I’ll take my Digital History course as an example, but I’m hoping to move all my courses to the same system, semester by semester. Repository setup Each course has two repositories: a main repo, mostly for my own use, but potentially of use to colleagues. These sites are small, with one file per content type – usually one each for syllabus, assignments, lectures, and some kind of “other” category. a website repo, containing one file per webpage – so instead of a single, an entry for each individual assignment. These are markdown files created by org-hugo-export-subtree. The website repo has two relevant branches: a master branch that contains the uncompiled source files, and a gh-pages branch that holds the Hugo-generated files that are actually served on the server. The gh-pages branch is checked out as a git worktree as described in the Hugo docs. It took me a little while to figure this out, but once you understand the concept of git worktrees it’s kind of fun to do things this way. For Digital History, I’m using the excellent docdock theme, and as they suggest, I’ve installed the theme as a git submodule. Workflow I do all my editing inside Emacs, in the original org-mode files that live in the “main” repo. Usually changes belong to one of two categories: “features” – planned updates to assignments, course materials, due dates, etc. “hotfixes” – corrections to errors, omissions, duplications in the course materials. I have to do these more often than I’d like to admit! In either case, the workflow should look like this: make the change in source (.org) text export change to Hugo inspect changes on local server if changes look good, commit them to master in the main site commit those changes to the web master run hugo -s in the website master dir; commit changes to gh-pages branch; push gh-pages to upstream. There are a lot of steps there! Fortunately we can automate or semi-automate them. I’ve left the first two as manual stages, because it’s smart to check changes before committing. However, I now make it a little easier by following Kaushal’s advice to run hugo server with the -p and --navigateToChanged switches. I now keep a pinned tab up in Firefox at localhost:xxxx (choose your own port with -p), and as soon as I export a page, that tab will update to the appropriate page. Fantastic! Makes checking my work much quicker. As for the rest, those steps are all done via 2 git post-commit hooks, one for each repo: in the main repo, I have a post-commit hook that checks if the commit was made on the master branch, and if so, then grabs the commit message & commit hash, then cd’s to the website directory & creates a new commit on master with references to the original commit. in the web repo, I have a second post-commit hook, which again checks if we’re on master, and if so, generates the website in the public directory (where we have the gh-pages site checked out as a git subtree), cd’s there, and commits to the gh-pages brnach. Then both branches are pushed to gh-pages I’m in a transitional period where I’m moving from self-hosted sites to gh-pages where possible. So I actually regenerate the site in another directory using hugo server -s -d myserver; I also have to update config.toml before doing this to fix a couple of paths. This is all pretty easy to do. Here’s the first, somewhat simpler script from the main repo: push_branch=master cur_branch=$(git rev-parse --abbrev-ref HEAD) cur_hash=$(git log -1 --pretty=%h) cur_msg=$(git log -1 --pretty=%B) web_dir=dh-website cur_dir=$PWD # only run if commit is to master if [ "$cur_branch" == "$push_branch" ]; then # if [ “$(git status -uno –porcelain)” ]; then # echo “uncommitted changes, can’t push”; # else # cd to website cd $web_dir # commit hcanges echo “committing changes to website master branch\n\n” git commit -a -m “push changes to website from titaniumbones/Digital-History#$cur_hash $cur_msg” cd $cur_dir echo “pushing master to origin” git push -f origin master fi And the slightly more baroque version from the website repo, which pushes to both gh-pages and to hackinghistory (shimano is just an alias for Note that I had to set up the hackinghistory repo as a non-bare repo with receive.denyCurrentBranch set to updateInstead. Thank you, stackoverflow!. push_branch=master cur_branch=$(git rev-parse --abbrev-ref HEAD) cur_hash=$(git log -1 --pretty=%h) cur_msg=$(git log -1 --pretty=%B) # only run if commit is to master if [ "$cur_branch" == "$push_branch" ]; then if [ "$(git status -uno –porcelain)" ]; then echo “uncommitted changes, can’t push”; else echo “pushing master to origin” git push -f origin master echo “Rebuilding Site” hugo -s ./ cd public && echo “changing env variables” && GIT_WORK_TREE="../.git/worktrees/public" && GIT_INDEX_FILE="../.git/worktrees/public/index"&& echo “adding all files\n\n” && git add –all echo “commitchanges to gh-pages\n\n” git commit -a -m “push changes to gh-pages from $cur_hash $cur_msg” echo “pushing gh-pages to origin” git push -f origin gh-pages && echo “successfully pushed gh-pages” cd .. sed ’s/baseURL = ""/baseURL = ""/' config.toml hugo -s ~/DH/dh-website/ -d ~/DH/dh-website/shimano/ # now shimano dir, shimano-deploy branch, shimano remote cd shimano && echo “changing env variables” && GIT_WORK_TREE="../.git/worktrees/shimano" && GIT_INDEX_FILE="../.git/worktrees/shimano/index"&& echo “adding all files\n\n” && git add –all echo “commit changes to shimano\n\n” git commit -a -m “push changes to shimano from $cur_hash $cur_msg” echo “pushing shimano-deploy to shimano” git push -f shimano shimano-deploy && echo “successfully pushed shimano” cd .. # rsync -azvbP ~/DH/dh-website/shimano/ shimano:/var/www/ mv config.toml fi fi

Troubling News

I’ve just learned that the being I thought was a cute dancing penguin is in fact an evil alien creature bent on destruction and domination. This creature has invaded a number of projects that I’m a part of, including Civic Tech, EDGI, and my classes “Technologies for Democracy” and “Digital History”. I can only assume that these projects have a hitherto-unrealized importance to the survival of the planet. Eternal Vigilance!

Org-Mode: Run Code Live in a Reveal slideshow with Klipse

Table of Contents The Basics: ox-reveal Running Code in Klipse Add Support for Third-Party Plugins Rewrite org-reveal-src-block Slide-based presentations are pretty useful and in any case de rigeur for contemporary lecturing, but it can be a REAL pain to switch form a slideshow to a code editor and then back again to a web browser to show the results of the code… Much better would be to have all the code embedded directly in your presentation. This also makes it easier to check over your code when you’re rewriting your lectures from year to year, and also helps with literate programming. It turns out that there are actually several solutions for doing this with Org. They are: directly embed JSBin in your slideshow Use RevealEditor for HTML, CSS, and JS that you want to render in a web page. Use the new klipse plugin for code that you want to evaluate (rather than render) Each method has advantages and disadvantages, and each requires a certain amount of setup in your config. In this post I’m talking about using Klipse because I’ve bene able to make it actually work! The Basics: ox-reveal To start with, you will need to install org-reveal as well as its dependencies (org-mode and reveal.js). I install org-reveal from github instead of from ELPA, in part because I need to make modifications to both it and reveal.js. <p> Follow the detailed instructions until you have a working export to reveal. Make sure that you have activated the “highlight” plugin in <code>org-reveal-plugins</code>, which you can do with <code>M-x customize-variable org-reveal-plugins</code>. None of the tricks described below will work without that plugin! </p> Running Code in Klipse Klipse is a web repl that supports a handful of languages, and can be extended to support more. Code is embedded in a CodeMirror instance and evaluated as you type (pretty cool). <p> Unfortunately CodeMirror and Reveal don’t get along very well so it’s not completely straightforward to embed Klipse directly into a reveal.js slideshow. The workaround is to replace a code-snippet tag with an <code>iframe</code> element containing the code in a form that klipse can read. Klipse’s author @viebel has kindly provided an example of how to do this <a href="">here</a>. </p> <p> The next trick is to set up org-reveal to use this snippet. After messing around for a bit it became obvious that org-mode’s standard <a href="">Export filters</a> were going to be pretty hard to use, so I decided to rewrite some of <code>org-reveal</code>’s functionality: </p> Add Support for Third-Party Plugins <div class="outline-text-3" id="text-org45d7e1d"> <p> Org-reveal has excellent support for reveal.js’s “build-in” plugins, but using third-party plugins requires you to use the <code>or-reveal~postamble</code> variable~, which didn’t seem right to me. So I <a href="">added a defcustom</a> and a few small generic tricks to support loading third-party plugins. Then I downloaded @viebel’s <code>klipse_reveal.js</code> script and saved it to the <code>plugin</code> folder of my <code>reveal.js</code> installation. Finally, I set org-reveal-external-plugins to <code>((klipse . "{src: '%splugin/klipse_reveal.js'}"))</code> using the customize-variable interface. </p> </div> Rewrite org-reveal-src-block <div class="outline-text-3" id="text-org35a8a7f"> <p> At this point, the plugin loads, but it couldn’t find the exported src blocks. As I mentioned above, I was unable to find a satisfactory solution using export filters, so in the end I just rewrote the function <code>org-reveal-src-block</code>, which ox-reveal uses to process src blocks. I decided to keep the original <code><pre><code></code> syntax and simply add on an additional <code><klipse-snippet</code> block, using the <code>display:none</code> CSS property for the old code block. This ensures compatibility with the reveal-editor solution (which isn’t currently working for me, but offers some other features. </p> <p> Here’s my modified function: </p> <div class="org-src-container"> <pre class="src src-emacs-lisp"><span style="color: #8c8c8c;">(</span><span style="color: #00ffff;">defun</span> <span style="color: #87cefa;">org-reveal-src-block</span> <span style="color: #8c8c8c;">(</span>src-block contents info<span style="color: #8c8c8c;">)</span> “Transcode a SRC-BLOCK element from Org to Reveal. CONTENTS holds the contents of the item. INFO is a plist holding contextual information." (if (org-export-read-attribute :attr_html src-block :textarea) (org-html–textarea-block src-block) (let* ((use-highlight (org-reveal–using-highlight.js info)) (lang (org-element-property :language src-block)) (caption (org-export-get-caption src-block)) (code (if (not use-highlight) (org-html-format-code src-block info) (cl-letf (((symbol-function ‘org-html-htmlize-region-for-paste) #‘buffer-substring)) (org-html-format-code src-block info)))) (frag (org-export-read-attribute :attr_reveal src-block :frag)) (code-attribs (or (org-export-read-attribute :attr_reveal src-block :code_attribs) "")) (label (let ((lbl (org-element-property :name src-block))) (if (not lbl) "" (format ” id="%s"" lbl)))) (klipsify (and (assoc ‘klipse org-reveal-external-plugins) (member lang ‘(“javascript” “ruby” “scheme” “clojure” “php”)))) ) (if (not lang) (format "<pre %s%s>\n%s</pre>" (or (frag-class frag info) " class="example"") label code) (format "<div %sclass="org-src-container">\n%s%s\n</div>%s" (if klipsify “style="display:none;" “ "") (if (not caption) "" (format "<label class="org-src-name">%s</label>" (org-export-data caption info))) (if use-highlight (format "\n<pre%s%s><code class="%s" %s>%s</code></pre>" (or (frag-class frag info) "") label lang code-attribs code) (format "\n<pre %s%s>%s</pre>" (or (frag-class frag info) (format ” class="src src-%s"" lang)) label code) ) (if klipsify (format "<klipse-snippet data-language="%s">%s</klipse-snippet>" lang code) "")))))) <p> As you can see, it now decides whether or not to “klipsify” the code aafter checking if the klipse plugin is loaded, and if it supports the language of the src block. Klipsifying code just means hiding the original code block and instead showing the klipse-snippet, which will be replaced by an iframe in javascript. I guess I could just do that work right in this same elisp function! Oh well, maybe next time. </p> <p> So far, this seems to work great, though I don’t have access to the DOM of the larger document ,which is sort of a drag. </p> <p> <b>[UPDATE <span class="timestamp-wrapper"><span class="timestamp">2016-12-11 Sun</span></span>]:</b> However, @viebel pointed out to me that you can access the parent frame with: </p> <div class="org-src-container"> <pre class="src src-js"><span style="color: #00ffff;">var</span> <span style="color: #eedd82;">d</span> = parent.document <p> Pretty cool! Note: klipse will hang if you try to type this in directly. It will only work if you pass the completed line in as part of the initial code block. </p> <p> Also, this is not a solution that works for rendering HTML pages. For this we need something else. I would like to use RevealEditor, but am not quite there yet (can’t get its code to load properly). So for now I’m embedding a JSBin instance with preoaded code – I just add an iframe tag directly in my org-mode file. It’s not a s pretty or robust, and I can’t see my code when I’m writing, which is too bad, but it’s still pretty useful. </p> <p> I would really like to hear what other people think! I’d especially like to hear about alternatives or improvements. </p> </div>

Using mu4e and org-mime together

I use org-mime in my grading system, to email my comments on student papers. One frustrating element has always been that messages sent by org-mime were never saved to my sent-mail folder. I realized just recently that this was because I had failed to set emacs’s mail-user-agent, which I now do in my initial mu4e setup: (setq mail-user-agent 'mu4e-user-agent) Now org-mime attempts to send messages using mu4e’s internal compose functions. Unfortunately some of the information passed by org-mime is in a format that mu4e~compose-mail doesn’t like, so I had to make some very slight changes to that function: diff --git a/mu4e/mu4e-compose.el b/mu4e/mu4e-compose.el index a24e74a..0c7ec3c 100644 --- a/mu4e/mu4e-compose.el +++ b/mu4e/mu4e-compose.el @@ -780,7 +780,14 @@draft message." ;; add any other headers specified (when other-headers -(message-add-header other-headers)) +(dolist (h other-headers other-headers) +(if (symbolp (car h)) (setcar h (symbol-name (car h)))) +(message-add-header (concat (capitalize (car h)) “: " (cdr h) “\n” )) +) +;; (dolist (h other-headers) +;; (message-add-header h) ) +;;(message-add-header other-headers) +) ;; yank message (if (bufferp yank-action) The commit is in its my org-mu4e-compose branch, contianing a couple of other fixes, and in its own branch on github, if prefer to pull from there. A patch has been submitted, we’ll see what dcjb thinks of it. With these changes, org-mime now works perfectly for me.

Sending HTML Mail with mu4e

John Kitchin has a terrific post detailing some configuration/improvements to mu4e that make it easier to send html mail. This original post is really great, but I ran into quite a bit of trouble following this advice. He uses a cool feature of mu4e, org-mu4e-compose-org-mode, which toggles the major mode of the message buffer between org-mode (when you’re in the message body) and mu4e-compose-mode (when you’re in the headers area). With a couple of custom functions, it’s easy to convert the org text to html and send a mime-multipart email from Emacs, which is quite convenient. If you add org-mu4e-compose-org-mode as a hook to mu4e-compose-mode, you can compose in html by default, which is great. Unfortunately, org-mu4e-compose-org-mode is deprecated on account of its instability, and while John doesn’t have any problems with it, for me it was unworkable. It turns out that this “mode” isn’t really a standard emacs mode at all – instead, it’s a sly workaround that trickily adds an internal function to the ’post-command-hook in the draft buffer and switches major modes based on position. This is a neat hack, but since the function invokes the major modes directly, setting thefunction as a hook to mu4e-compose-mode leads to some funky, inadvertent looping effects. On my machine, for some reason, those effects send the underlying message-send function crazy, and instead of sending directly, I get this amazingly annoying question: Already sent message via mail; resend? (y or n) y On its own, that is already annoying; but worse, the sent message doesn’t get saved to my Sent folder. Instead, it’s lost completely. Anyway, after fruitless hours of paying around with this, I realized that the problem could be fixed by adding a new hook to the mu4e compose functions (rather than to the compose mode). I’ve submitted those changes as a pull request and hopefully they will be accepted; if not, though, feel free to navigate back to my branch and pull/install mu from there. With those small changes, I now have frictionless html email working very quickly within emacs, using this small bit of code. It is at least 90% stolen: ;; this is stolen from John but it didn't work for me until I ;; made those changes to mu4e-compose.el (defun htmlize-and-send () "When in an org-mu4e-compose-org-mode message, htmlize and send it." (interactive) (when (member 'org~mu4e-mime-switch-headers-or-body post-command-hook) (org-mime-htmlize) (org-mu4e-compose-org-mode) (mu4e-compose-mode) (message-send-and-exit))) ;; This overloads the amazing C-c C-c commands in org-mode with one more function ;; namely the htmlize-and-send, above. (add-hook ‘org-ctrl-c-ctrl-c-hook ‘htmlize-and-send t) ;; Originally, I set the `mu4e-compose-mode-hook’ here, but ;; this new hook works much, much better for me. (add-hook ‘mu4e-compose-post-hook (defun do-compose-stuff () “My settings for message composition." (org-mu4e-compose-org-mode))) It feels great to have gotten this far. There are still some small things I’d like to be able to improve; and I think I would like to add a few wrapper functions and keybindings to my setup, but for now I’m pretty efficient. Still missing: a better HTML viewing interface! right now, html mails render as mostly text – it would be nice to have a rendered html message by default in emacs. This is an issue several times a day when I get promotional emails from organizations I work with – usually the html part is really important. I can access these in the browser but it’s comparatively awkward. a way to forward these html emails to someone intact – right now, the html parts are discarded. No idea how hard it would be to do this. Thanks John and Dirk-Jan for these great tools!

It’s Time to Finally Take Encryption Seriously!

Table of Contents Encryption Review Messaging Email We have known for a long time that the American intelligence agencies don’t flinch at violating our privacy. Edward Snowden’s revelations have forced some small reforms at the NSA and elsewhere; but those small improvements will doubtless be rolled back, and XKeyscore, PRISM and, ECHELON will be replaced by even more invasive and dangerous technologies. They will be controlled by a government with strong ties to White Nationalists, militias, and various hate groups. In other times and places, governments have circulated death lists of dissenters to paramilitary organizations. I don’t think that’s likely in the USA; but I also think that as citizens, we need to act to protect ourselves from the abuse of power. The only effective tool against government surveillance is end-to-end encryption of communication. Even encryption is likely to be inadequate against a targeted attack by the NSA’s supercomputers and tactical operations; but when it comes to the mass collection of data, properly-practiced encryption protects you from being targeted for more detailed analysis. As a by-product, encrypting your communications makes it a little bit harder (but not much!) for corporations to track your preferences, and also adds a little bit of protection from other malicious agents. The basic building blocks of encrypted communication have been available for a long time, but until recently encryption was clumsy and error-prone. Encryption of email is still somewhat challenging. In the last year, though, two excellent solutions, and one adequate one, have emerged for messaging. I’ll talk about all these in a minute, but first let’s quickly review how encryption works. Encryption Review Almost all contemporary encryption is “dual-key” – each person in a conversation has both a private and a public “key”. If I want to send you a message, I encrypt it with your public key. After that, even I can’t read my message – only you can, with your private key, which you don’t show to anyone. <p> This sounds simple, but it is actually quite difficult to implement encrypted communication in a simple, easy-to-understand interface. Software developers have struggled with this for a long time. It’s worth reading a little more about this: </p> <ul class="org-ul"> <li> <a href="">The EFF’s Surveillance Self-Defense Guide</a> is a comprehensive introduction to encryption practice. You should read it. </li> <li> <a href="">The FSF also has an Email Self-Defense Guide</a>. Specific to email, but still helpful, and quite a bit shorter. </li> <li> The EFF used to have a <a href="">Secure Messaging Scorecard</a>. The first version has been deprecated, but a new one will be launched soon; read it when it comes out. It will likely supersede much of this information. </li> </ul> Messaging The messaging landscape has been changing rapidly. After a long delay, there is now really good encryption available, but some of the interfaces are a little confusing or misleading. <ul class="org-ul"> <li> <b><a href="">Signal</a></b> allows you to send text messages and make phone calls with full, end-to-end, dual-key encryption. You should switch to it immediately. You won’t be able to make encrypted phone calls over the regular phone network – you will need to use data instead. </li> <li> As of April, <b><a href="">WhatsApp</a></b> now offers dual-key encryption by default, including a key-verification interface. Among the mainstram apps, WhatsApp is probably the best option, but <a href="">it’s not perfect</a>, and Signal is better if you can use it. </li> <li> In June, <b><a href="">Facebook Messenger</a></b> enabled end-to-end encryption. Encryption is turned off by default and has to be turned on for every conversation, so this is <b>not</b> as good as WhatsApp, but if your social media life mostly moves through Facebook, you <b>can</b> enable encryption, and it’s worth doing. </li> <li> Google’s new <a href="">Allo</a> messenging app also permits end-to-end encryption, but it’s not on by default, and it’s a bit confusing to use. Signal is still ldefinitely preferable. </li> </ul> Email Email is tricky. For most people, Thunderbird plus EnigMail is probably the best option, but it is confusing to use and it also “leaks” information (as will always be the case for email sent through traditional channels). Various projects to fix email were announced in the wake of the Snowden revelations, but the onse I know about have mostly stopped development. It’s a hard nut to crack. <p> Still: you should aim to encrypt a relatively high percentage of your emails if you can. It’s not good enough to just encrypt a few; the important ones need to be masked by the trivial ones. And encrypting your emails also helps to protect the journalists, activists, ando thers who need to encrypt their communication <i>for their own protection</i>. We encrypt not just for ourselves, but to preserve essential freedoms for others. </p> <p> Over the next few months I hope to transition to encryption in most of my messaging; that means that, if you want to be in touch with me online, you’ll need to start figuring this stuff out too. Let me know what I can do to help! </p>

Using the Sikkim Website to keep in touch

Table of Contents Getting Started Location Data Featured Image Showing a post on the “Our Journey” Page OK, we’re getting down to the wire. This post is for the Sikkim trip students…. I know most of you will keep in touch with friends and family via Facebook & other social media. However, I’m hoping that we will also use the group website as a way to keep in touch while we’re gone. As you know, I’ve set the site up as a major part of our intellectual output for this project, and yo all have accounts that should let you write and publish blog posts & webpages. Here are just a few short tips on how to use the site. I’m hoping those of you who are familiar with WordPress can help the others. Getting Started Use your username and password to get into the site at the admin/login page. To create a new journal entry, go to Posts/New (or tap the + button in the top menu. Compose your message in the editor box, add pictures using the “add media’ button in the editor toolbar, press ”Publish“, and you’re done. Location Data Since we’re thinking of this as “place-based” education, location data is pretty important to us. You can add three kinds of location information to your post (underneath the editor box): <dl class="org-dl"> <dt> GPX Tracks </dt> <dd> These are recordings of where you’ve been (discussed in my last post). If you attach one of these in the “GPX File” upload box, it will display in your post (but not in the list of all the posts…). We’ll use these a lot when we’re in Sikkim. </dd> <dt> KML FIles </dt> <dd> There’s a lot of stuff these can do, but the main reason I included them is that in some cases they can be easier to export than gpx tracks. </dd> <dt> Co-ordinates </dt> <dd> You can use this to indicate where you are in the moment that you’re writing your post. This is what will show up in the “archive” and “our journey” pages (yo can look at the test pages for examples of how it will look). </dd> </dl> <p> I strongly encourage you to use the geodata options here! It’ll help our site out a lot. </p> Featured Image You’ll notice that each post has one special image associated with it, that is used as a packground image in a couple of places. This is the “featured image”, which you set in a box on the lower right of the writing interface page (or, if you’re using your phone, just really far down in the interface). I’d really like it if every post had a featured image, so try to choose a good one! Showing a post on the “Our Journey” Page “Our Journey” is, as it were, the “official” record of our trip. Each day one or two of us will write a special post about the day’s events; it’s expected to be perhaps a little more substantive or public-facing than your regular travel diary. You can move a post into the “Our Journey” feed by adding the “Our Journey” category – in the “Category” box. <p> Youu may also notice the “Tags” box – we an talk later about how to make use of tags on the site. </p> <p> See yo all in India! </p>

Android Mapping Apps

Table of Contents Offline Map Display Data Storage GPS Tracking Getting your data out REVISED For our trip to Sikkim, I would like students to have a way to map our progress on their phones so that it can be displayed on our website. I’ve spent a bit of time looking at various Android apps (I don’t have an iPhone) and have some thoughts about how to do this; here are my notes, intended mostly for the students but also potentially for other people looking to do something similar. Offline Map Display Most mapping applications rely on a continuous data feed, which will not be available to us in Sikkim. So, what applications offer offline viewing? <p> <a href="">Google Maps</a> allows downloading of areas for offline use. <a href="">Here are the official instructions for doing so.</a> The Sikkim area takes about 500mb of space so be aware of that. </p> <p> <a href="">OSMAnd</a> is an interesting alternative. It is mostly designed for offline use. If you buy the <a href="">countour Lines Plugin</a> for $1.99, you can add topographic map contour lines to your map, which is pretty cool. I kind of love it myself, and it’s what I’ll be using. The interface is perhaps a little less intuitive than the Google Maps interface, but I’m getting used to it. You will need to download the world base map, as well as the detailed map for all of India (if you have enough space on your phone) or the information for Sikkim only (if you don’t have enough room). The contour map is a separate download, and to be able to use it, you will need to purchase the plugin linked to above. </p> <p> The free version of the app allows only 7 downloads in total, but we shouldn’t need more than that for our trip. I myself bought the <a href="">full paid version</a> but that shouldn’t be necessary IIUC. </p> Data Storage If your phone supports the use of a removable microSD card (most do, mine doesn’t), I would highly recommend buying one. You can get an 8gb card for $4.99 at Canada Computers on College St, and that will give you plenty of space for everything you want to put on it. Buy it before you install the app, so that you can choose to store your data on the card rather than the phone’s internal memory. GPS Tracking It’s nice to have a map on your phone, but most of the time you won’t be using the map at all. The real reason for this post is that we’d like to place at least one of you on “backup tracking” duty every day of the trip. For this you need some kind of tracking software, which can generate files in either .gpx or .klm format; and you also need a way to get the data out of the phone and into one of our tablets for writing. <p> There are a number of apps that let you do this. OSMAnd+ (see above) lets you create gpx tracks, though it doesn’t let you rename or share them direcly, which is a bit of a pain. Here are some alternatives: </p> <p> <a href="">GPS Logger</a> is a very simple, lightweight app that only records gpx/kml tracks for you. It doesn’t display a map or anything. Its main advantage is that it doesn’t use much energy compared to other apps. </p> <p> <a href="">Geo tracker</a> is a pretty user-friendly app. Unfortunately it can’t make use of OSMAnd’s downloaded maps so I’m not sur what the display will look like when you don’t have any data. </p> <p> There are <a href=""></a>]][zillions of other apps]]. Hopefully one of the above works for you but if not the Open Stree Map wiki is likely your best bet. </p> Getting your data out REVISED Unfortunately our tablet doesn’t support NFC!! So you will need to transfer your files manually onto a USB thumb drive. To do this you will need a file manager. There are lots of them; you can find lists here and here. I don’t like any of them all that much but they should work, at least a little. If you have a brand-new version of Android you can also use the built-in file manager. In general it’s something of a pain, hopefully we will only be doing that for a few days when we’re completely off the grid. <p> <b>NEW:</b> I had a lot of trouble with most of the file managers, and ended up installing <a href="">ES File Explorer</a>. Now I’m able to move files easily onto an external thumb drive attached to my phone with an On The Go cable (I’m bringing 2 so we should be fine on the trip). Note that some users have reported concerns about personal data being stored on Baidu, the Chinese Google. If you have anything on your phone that you don’t want the Chinese government to know about, you may want to consider another option. </p> <p> However: ES File Explorer is <b>MUCH</b> less painful than other options. With “ES Save To”, you can save files directly to an external USB drive attached to your phone, which can then be transferred easily to a tablet or compter. </p> <p> It’s still a pain to locate your .gpx files. </p>

PDF Tools Post amended

I’ve updated my older post on PDF Tools with slightly cleaner code, for anyone who’s using this method of extracting annotations!


I’ve become pretty addicted to the Elfeed RSS reader. However, I haven’t made much use of its tagging capabilities until recently. I noticed that, whenever I thought I might want to keep track of the content of a blog, I would mark it “unread” after I’d read it, to make sure I didn’t lose it in the deep mists of time. This worked for the first 4 months or so, but now my list of unreads is growing enormously. So I wanted to have a “starred” tag which would do this work for me. I also wanted a keybinding that would apply the tag right away. The code below does that for me. (eval-after-load 'elfeed-search '(define-key elfeed-search-mode-map (kbd "<tab>") 'mwp/elfeed-star)) (defun mwp/elfeed-star () “add a star tag to marked” (interactive) (elfeed-search-tag-all (list starred)) ) (defun mwp/elfeed-star () “Apply TAG to all selected entries." (interactive ) (let* ((entries (elfeed-search-selected)) (tag (intern “starred”))) <span style="color: #707183;">(</span><span style="color: #a020f0;">cl-loop</span> for entry in entries do <span style="color: #707183;">(</span>elfeed-tag entry tag<span style="color: #707183;">))</span> <span style="color: #707183;">(</span>mapc #'elfeed-search-update-entry entries<span style="color: #707183;">)</span> <span style="color: #707183;">(</span><span style="color: #a020f0;">unless</span> <span style="color: #707183;">(</span>use-region-p<span style="color: #707183;">)</span> <span style="color: #707183;">(</span>forward-line<span style="color: #707183;">))))</span> I also wanted a visual cue to tell me the tagging had been successful – now easy to do, thanks to a very recent commit. I filed an issue as a question – can I do this? – and within 24 hours skeeto had added the functionality. First you have to define an appropriate face for the tag: (defface elfeed-search-starred-title-face '((t :foreground "#f77")) "Marks a starred Elfeed entry.") And then simply add an entry to elfeed-search-face-alist. You can do this via Customize, or follow the instructions in the README, which instruct you to (push '(starred elfeed-search-starred-title-face) elfeed-search-face-alist). All of this works great for me, and simplifies things quite a bit.

Note Taking with PDF Tools

NOTE: This post has been modified as of 2015-11-22 Sun – the new code is a little cleaner, and I think the discussion a little fuller. Almost all of my job-related reading is now done on a screen. There are still disadvantages – I find it much harder to concentrate when reading online – but in other ways it is markedly more convenient. In particular, it is now much easier to assemble quotations from sources; and now that I’ve found PDF Tools, it has become even easier. I’ve just started to use it to extract annotations from my PDF’s, and it works much better than the lousy command-line hack I was using previously. As we’re mid-semester, most of my reading is for classes I teach. My current workflow is as follows: Assemble the relevant readings in a Dropbox-synced directory (ClassName/Readings) Using Repligo Reader (apparently no longer available in the app store?), highlight the passages I’m interested in. execute code block (see below) to insert org headings with all highlights from one or more pdfs Assemble reveal.js lecture presentation around those highlights, using org-reveal or Pandoc. Activating PDF Tools Begin by installing pdf-tools and org-pdfview from ELPA with package-list~packages or package-install. <p> Then make sure they are activated by adding these lines in your init file: </p> <div class="org-src-container"> <pre class="src src-emacs-lisp"><span style="color: #707183;">(</span>pdf-tools-install<span style="color: #707183;">)</span> (eval-after-load ‘org ‘(require ‘org-pdfview)) (add-to-list ‘org-file-apps ‘("\.pdf\'" . org-pdfview-open)) (add-to-list ‘org-file-apps ‘("\.pdf::\([[:digit:]]+\)\'" . org-pdfview-open)) Switching to PDF Tools for annotating and extracting PDF’s Last month Penguim proposed some changes in a pull request, that export annotations as a set of org headlines. It’s potentially very interesting but not quite what I want to do, so I modified this code. pdf-annot-markups-as-org-text extracts the text of an annotation (stored as the subject attribute in an alist), and also generates a link back to the page in the pdf. mwp/pdf-multi-extract is just a helper function that makes it easier to construct elisp source blocks the way I’m used to doing: <div class="org-src-container"> <pre class="src src-emacs-lisp"><span style="color: #b22222;">;; </span><span style="color: #b22222;">modified from </span> (defun mwp/pdf-multi-extract (sources) “Helper function to print highlighted text from a list of pdf’s, with one org header per pdf, and links back to page of highlight." (let ( (output "")) (dolist (thispdf sources) (setq output (concat output (pdf-annot-markups-as-org-text thispdf nil level )))) (princ output)) ) ;; this is stolen from (defun pdf-annot-edges-to-region (edges) “Attempt to get 4-entry region (LEFT TOP RIGHT BOTTOM) from several edges. We need this to import annotations and to get marked-up text, because annotations are referenced by its edges, but functions for these tasks need region." (let ((left0 (nth 0 (car edges))) (top0 (nth 1 (car edges))) (bottom0 (nth 3 (car edges))) (top1 (nth 1 (car (last edges)))) (right1 (nth 2 (car (last edges)))) (bottom1 (nth 3 (car (last edges)))) (n (safe-length edges))) ;; we try to guess the line height to move ;; the region away from the boundary and ;; avoid double lines (list left0 (+ top0 (/ (- bottom0 top0) 2)) right1 (- bottom1 (/ (- bottom1 top1) 2 ))))) (defun pdf-annot-markups-as-org-text (pdfpath &optional title level) “Acquire highligh annotations as text, and return as org-heading” (interactive “fPath to PDF: “) (let* ((outputstring "") ;; the text to be returned (title (or title (replace-regexp-in-string "-" ” “ (file-name-base pdfpath )))) (level (or level (1+ (org-current-level)))) ;; I guess if we’re not in an org-buffer this will fail (levelstring (make-string level ?*)) ;; set headline to proper level (annots (sort (pdf-info-getannots nil pdfpath) ;; get and sort all annots ‘pdf-annot-compare-annotations)) ) ;; create the header (setq outputstring (concat levelstring ” Quotes From “ title "\n\n”)) ;; create heading <span style="color: #b22222;">;; </span><span style="color: #b22222;">extract text</span> <span style="color: #707183;">(</span>mapc <span style="color: #707183;">(</span><span style="color: #a020f0;">lambda</span> <span style="color: #707183;">(</span>annot<span style="color: #707183;">)</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">traverse all annotations</span> <span style="color: #707183;">(</span><span style="color: #a020f0;">if</span> <span style="color: #707183;">(</span>eq 'highlight <span style="color: #707183;">(</span>assoc-default 'type annot<span style="color: #707183;">))</span> <span style="color: #707183;">(</span><span style="color: #a020f0;">let*</span> <span style="color: #707183;">((</span>page <span style="color: #707183;">(</span>assoc-default 'page annot<span style="color: #707183;">))</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">use pdf-annot-edges-to-region to get correct boundaries of highlight</span> <span style="color: #707183;">(</span>real-edges <span style="color: #707183;">(</span>pdf-annot-edges-to-region <span style="color: #707183;">(</span>pdf-annot-get annot 'markup-edges<span style="color: #707183;">)))</span> <span style="color: #707183;">(</span>text <span style="color: #707183;">(</span><span style="color: #a020f0;">or</span> <span style="color: #707183;">(</span>assoc-default 'subject annot<span style="color: #707183;">)</span> <span style="color: #707183;">(</span>assoc-default 'content annot<span style="color: #707183;">)</span> <span style="color: #707183;">(</span>replace-regexp-in-string <span style="color: #8b2252;">"\n"</span> <span style="color: #8b2252;">" "</span> <span style="color: #707183;">(</span>pdf-info-gettext page real-edges nil pdfpath<span style="color: #707183;">)</span> <span style="color: #707183;">)</span> <span style="color: #707183;">))</span> <span style="color: #707183;">(</span>height <span style="color: #707183;">(</span>nth 1 real-edges<span style="color: #707183;">))</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">distance down the page</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">use pdfview link directly to page number</span> <span style="color: #707183;">(</span>linktext <span style="color: #707183;">(</span>concat <span style="color: #8b2252;">"[[pdfview:"</span> pdfpath <span style="color: #8b2252;">"::"</span> <span style="color: #707183;">(</span>number-to-string page<span style="color: #707183;">)</span> <span style="color: #8b2252;">"++"</span> <span style="color: #707183;">(</span>number-to-string height<span style="color: #707183;">)</span> <span style="color: #8b2252;">"]["</span> title <span style="color: #8b2252;">"]]"</span> <span style="color: #707183;">))</span> <span style="color: #707183;">)</span> <span style="color: #707183;">(</span><span style="color: #a020f0;">setq</span> outputstring <span style="color: #707183;">(</span>concat outputstring text <span style="color: #8b2252;">" ("</span> linktext <span style="color: #8b2252;">", "</span> <span style="color: #707183;">(</span>number-to-string page<span style="color: #707183;">)</span> <span style="color: #8b2252;">")\n\n"</span><span style="color: #707183;">))</span> <span style="color: #707183;">)))</span> annots<span style="color: #707183;">)</span> outputstring <span style="color: #b22222;">;; </span><span style="color: #b22222;">return the header</span> <span style="color: #707183;">)</span> ) Using in Org with a Source Block Now it’s more or less trivial to quickly generate the org headers using a source block: <pre class="example"> #+BEGIN_SRC elisp :results output raw :var level=(1+ (org-current-level)) (mwp/pdf-multi-extract ‘( “/home/matt/HackingHistory/readings/Troper-becoming-immigrant-city.pdf” “/home/matt/HackingHistory/readings/historical-authority-hampton.pdf”)) #+END_SRC <p> And the output gives something like </p> <p> #+BEGIN_EXAMPLE </p> Quotes From Troper becoming immigrant city Included in the Greater Toronto Area multiethnic mix are an estimated 450,000 Chinese, 400,000 Italians, and 250,000 African Canadians, the largest component of which are ofCar- ibbean background, although a separate and distinct infusion of Soma- lis, Ethiopians, and other Africans is currently taking place. (Troper becoming immigrant city, 3) <p> Although Toronto is Canada’s leading immigrant-receiving centre, city officials have neither a hands-on role in immigrant selection nor an official voice in deciding immigration policy. In Canada, immigration policy and administration is a constitutional responsibility of the fed- eral government, worked out in consultation with the provinces. (<a href="">Troper becoming immigrant city</a>, 4) </p> <p> #+END_EXAMPLE </p> Alternative: Temporary buffer with custom link type An alternative workflow would be to pop to a second, temporary buffer and insert the annotations there; one could do this with a custom link type. PDF-Tools already has a mechanism for listing annotations in a separate buffer, but it’s not designed for quick access to all annotations at once. Anyway, here’s one way to do this; I’m not really using it at the moment. <div class="org-src-container"> <pre class="src src-emacs-lisp"><span style="color: #707183;">(</span>org-add-link-type <span style="color: #8b2252;">"pdfquote"</span> 'org-pdfquote-open 'org-pdfquote-export<span style="color: #707183;">)</span> (defun org-pdfquote-open (link) “Open a new buffer with all markup annotations in an org headline." (interactive) (pop-to-buffer (format "Quotes from %s" (file-name-base link))) (org-mode) (erase-buffer) (insert (pdf-annot-markups-as-org-text link nil 1)) (goto-char 0) ) (defun org-pdfquote-export (link description format) “Export the pdfview LINK with DESCRIPTION for FORMAT from Org files." (let* ((path (when (string-match "\(.+\)::.+" link) (match-string 1 link))) (desc (or description link))) (when (stringp path) (setq path (org-link-escape (expand-file-name path))) (cond ((eq format ‘html) (format "<a href="%s">%s</a>" path desc)) ((eq format ‘latex) (format "\href{%s}{%s}" path desc)) ((eq format ‘ascii) (format "%s (%s)" desc path)) (t path))))) (defun org-pdfquote-complete-link () “Use the existing file name completion for file. Links to get the file name, then ask the user for the page number and append it." (replace-regexp-in-string "^file:" “pdfquote:" (org-file-complete-link))) <p> I’ve also added two bindings to make highlighting easier from the PDF buffer: </p> <div class="org-src-container"> <pre class="src src-emacs-lisp"><span style="color: #707183;">(</span>eval-after-load 'pdf-view '<span style="color: #707183;">(</span>define-key pdf-view-mode-map <span style="color: #707183;">(</span>kbd <span style="color: #8b2252;">"M-h"</span><span style="color: #707183;">)</span> 'pdf-annot-add-highlight-markup-annotation<span style="color: #707183;">))</span> (eval-after-load ‘pdf-view ‘(define-key pdf-view-mode-map (kbd "<tab>") ‘pdf-annot-add-highlight-markup-annotation)) <p> All of this is getting me very close to using Emacs for all my PDF work. I am doing maybe 50% of my PDF work in Emacs instead of on my tablet. It’s incredibly convenient, although I still find it a little harder to concentrate on my laptop than on the tablet (for reasons ergonomic, optical, and psychological). Here are the remaining papercuts from my perspective: </p> <ul class="org-ul"> <li> Highlighting text with the mouse is more awkward and less intimate than using my fingertip on a laptop. I often find mouse movement a little awkward in Emacs, but pdf-view purposely relies on the mouse for movement (for good reasons). </li> <li> Scrolling in pdf-view is also a bit awkward, and there’s no “continuous” mode as one might find in Evince or acroread. Again, I often find scrolling an issue in Emacs, so this might not be so easy to fix. </li> <li> Finally, the laptop screen is just harder on my eyes than my high-res tablet. pdf-view hasa “midnight mode” which makes it a little easier to read, but it’s not quite enough. </li> </ul> <p> So, for the time being I will probably do much of my reading on the tablet. But for short pieces and for review (e.g., papers that I’m reading for the third year in a row in a graduate seminar…) PDF Tools is now my main interface. Which is pretty sweet. </p> Todo UPDATE: I would like to extend the pdfview link type (in org-pdfview) to permit me to specify the precise location of an annotation, so I can jump precisely to that part of the page. This has now been done and the code above has been updated to reflect the new syntax. <p> <b>UPDATE:</b> <del>Also, now that I think about it, it might be interesting to just have a link type that pops up a temporary buffer with all of the annotations; I could then cut and paste the annotations into the master document. This might be even more convenient.</del> OK, I’ve implemented this, see above! </p> <p> <b>Update <span class="timestamp-wrapper"><span class="timestamp">2015-11-22 Sun</span></span>:</b> I’ve cleaned up some of the code, and added a bit more commentary at the end. </p>

Mailing subtrees with Attachments

This is pretty exciting; I’ve figured out how to quickly send org-mode subtrees as MIME-encoded emails. That means that, essentially, I can write in org, as plain text, and very quickly export to HTML, add attachments, and send. The exciting part about this for me is that it should streamline my communications with students, while also letting me stay in Org and keep my records in order. Let’s walk through the process. Use-case For the moment, I still use Thunderbird as my primary MUA. It’s pretty easy to use, minimal configuration compared to all things Emacs, and if something goes wrong with it I don’t have to quit Emacs (!), just Thunderbird. <p> In some cases, though, Thunderbird makes for an awkward workflow. That’s certainly the case for grading, whih has many, poorly-integrated elements. To mark an assignment I need to: </p> <ul class="org-ul"> <li> log in to Blackboard (in Firefox; I wonder if I could do that in Emacs?) </li> <li> download the set of student papers (one at a time, 2 clicks per paper) to <code>Downlads</code> </li> <li> move papers to a directory (usually ~/COURSENAME/Grading/ASSIGNMENTNAME ) </li> <li> Read papers in LigreOffice, comment inline </li> <li> record mark in Libreoffice Calc spreadsheet </li> <li> email paper back to student with comments </li> <li> upload marks back into Blackboard </li> <li> find a place to archive the student paper in case I need it later, e.g. for a contested grade. </li> </ul> <p> The while process basically sucks. I spend maybe 20% of my time fussing with paths and mouse clicks and email addresses. So I am experimenting with moving as much of this process into Emacs. So far, I don’t think there’s any way at all to bulk-download the papers – that sucks, but I can live with it I guess (I have to!). So I start the optimization at the point where I have all my papers ready to go in a subdir. </p> Org-mime Org-mime is the library that allows org buffers and other elements to be quickly converted to HTML and prepared for multi-part messaging. Load it and set it up (see <div class="org-src-container"> <pre class="src src-emacs-lisp"><span style="color: #b22222;">;; </span><span style="color: #b22222;">enable HTML email from org</span> (require ‘org-mime) ;; setup org-mime for wanderlust ;; (setq org-mime-library ‘semi) ;; or for gnus/message-mode (setq org-mime-library ‘mml) ;; easy access to htmlize in message-mode (add-hook ‘message-mode-hook (lambda () (local-set-key "\C-c\M-o" ‘org-mime-htmlize))) ;; uncomment this to use the org-mome native functions for htmlizing. ;; (add-hook ‘org-mode-hook ;; (lambda () ;; (local-set-key “\C-c\M-o” ‘org-mime-org-buffer-htmlize))) ;; uncomment to displyay src blocks with a dark background ;; (add-hook ‘org-mime-html-hook ;; (lambda () ;; (org-mime-change-element-style ;; “pre” (format “color: %s; background-color: %s; padding: 0.5em;" ;; "#E6E1DC” “#232323”)))) ;; pretty blockquotes (add-hook ‘org-mime-html-hook (lambda () (org-mime-change-element-style <span style="color: #8b2252;">"blockquote"</span> <span style="color: #8b2252;">"border-left: 2px solid gray; padding-left: 4px;"</span><span style="color: #707183;">)))</span> Fix htmlization Upstream org-mime-htmlize unfortunately can’t be called uninteractively (bummer!), so we have to rewrite it to make programmatic calls work properly. I found the solution Emacs Stackexchange. <div class="org-src-container"> <pre class="src src-emacs-lisp"><span style="color: #707183;">(</span><span style="color: #a020f0;">defun</span> <span style="color: #0000ff;">org-mime-htmlize</span> <span style="color: #707183;">(</span><span style="color: #228b22;">&optional</span> arg<span style="color: #707183;">)</span> “Export a portion of an email body composed using </span><span style="color: #008b8b;">mml-mode</span><span style="color: #8b2252;">' to</span> <span style="color: #8b2252;">html using org-mode’. If called with an active region only export that region, otherwise export the entire body." (interactive “P”) (require ‘ox-org) (require ‘ox-html) (let* ((region-p (org-region-active-p)) (html-start (or (and region-p (region-beginning)) (save-excursion (goto-char (point-min)) (search-forward mail-header-separator) (+ (point) 1)))) (html-end (or (and region-p (region-end)) ;; TODO: should catch signature… (point-max))) (raw-body (concat org-mime-default-header (buffer-substring html-start html-end))) (tmp-file (make-temp-name (expand-file-name “mail” temporary-file-directory))) (body (org-export-string-as raw-body ‘org t)) ;; because we probably don’t want to export a huge style file (org-export-htmlize-output-type ‘inline-css) ;; makes the replies with “>“s look nicer (org-export-preserve-breaks org-mime-preserve-breaks) ;; dvipng for inline latex because MathJax doesn’t work in mail (org-html-with-latex ‘dvipng) ;; to hold attachments for inline html images (html-and-images (org-mime-replace-images (org-export-string-as raw-body ‘html t) tmp-file)) (html-images (unless arg (cdr html-and-images))) (html (org-mime-apply-html-hook (if arg (format org-mime-fixedwith-wrap body) (car html-and-images))))) (delete-region html-start html-end) (save-excursion (goto-char html-start) (insert (org-mime-multipart body html (mapconcat ‘identity html-images "\n”)))))) Actually perform the export! These functions are crude helpers that gather extra information about the org subtree, of which org-mime is unaware. <ul class="org-ul"> <li> <code>mwp-org-get-parent-headline</code> traverses the tree to the ancestor headline, because that’s what I want to set the subject to. </li> <li> <code>mwp-org-attachment-list</code> is stolen directly from the Gnorb package, which looks cool, awesome ,and kinda complex; it just iterates through a subtree’s attachments and grabs URLs. </li> <li> <code>mwp-send-subtree-with-attachments</code> performs the export and is bound to <code>C-c M-o</code> </li> </ul> <p> So, if I want to mail a subtree, I just <code>C-c M-o</code> and I’m almost done – the html mail is ready to go, and all org attachments are also attached to the email. </p> <p> Note there are some real weaknesses here: <code>mwp-org-get-parent-headline</code> actually gets the top-level <i>ancestor</i> – which only happens to be what I want right now. Better would be to use org-element to locate the parent (and other headline attributes) directly, but I’m not sure how to do that. </p> <p> Similarly, the initial greeting is generated from the current headline value – so this only works because I name my subtrees after the addressee (which I only do because of my use case). </p> <div class="org-src-container"> <pre class="src src-emacs-lisp"><span style="color: #707183;">(</span><span style="color: #a020f0;">defun</span> <span style="color: #0000ff;">mwp-org-get-parent-headline</span> <span style="color: #707183;">()</span> “Acquire the parent headline & return." (save-excursion (org-mark-subtree) (re-search-backward "^\* “) (nth 4 (org-heading-components)))) (defun mwp-send-subtree-with-attachments () “org-mime-subtree and HTMLize” (interactive) (org-mark-subtree) (let ((attachments (mwp-org-attachment-list)) (subject (mwp-org-get-parent-headline))) (insert “Hello “ (nth 4 org-heading-components) ”,\n”) (org-mime-subtree) (insert "\nBest,\nMP.\n”) (message “subject is” ) (message subject) ;;(message-to) (org-mime-htmlize) ;; this comes from gnorb ;; I will reintroduce it if I want to reinstate questions. ;; (map-y-or-n-p ;; ;; (lambda (a) (format “Attach %s to outgoing message? “ ;; ;; (file-name-nondirectory a))) ;; (lambda (a) ;; (mml-attach-file a (mm-default-file-encoding a) ;; nil “attachment”)) ;; attachments ;; '(“file” “files” “attach”)) ;; (message “Attachments: %s” attachments) (dolist (a attachments) (message “Attachment: %s” a) (mml-attach-file a (mm-default-file-encoding a) nil “attachment”)) (message-goto-to) )) ;; add a keybinding for org-mode (add-hook ‘org-mode-hook (lambda () (local-set-key "\C-c\M-o” ‘mwp-send-subtree-with-attachments))) ;; stolen from gnorb; finds attachments in subtree (defun mwp-org-attachment-list (&optional id) “Get a list of files (absolute filenames) attached to the current heading, or the heading indicated by optional argument ID." (when (featurep ‘org-attach) (let* ((attach-dir (save-excursion (when id (org-id-goto id)) (org-attach-dir t))) (files (mapcar (lambda (f) (expand-file-name f attach-dir)) (org-attach-file-list attach-dir)))) files))) Contacts That’s a good start, but there are still some steps to make this truly convenient. For instance, I certainly don’t want to type in students’ email addresses by hand. So I imported my contacts from thunderbird to org-contacts. This was a pain – the process was Thunderbird → Gmail (via gsync plugin) → vcard (via gmail export) → org-contacts (via Titus’s python importer). I wish there was a CSV importer for org-contacts; probably this would be easy to write but I’m so slooooowwww at coding. My org contacts live in GTD/, which is set in Customize, and org reads them on startup with this line <div class="org-src-container"> <pre class="src src-emacs-lisp"><span style="color: #707183;">(</span><span style="color: #a020f0;">require</span> '<span style="color: #008b8b;">org-contacts</span><span style="color: #707183;">)</span> <p> With this single line, org-contacts now provides <kbd>TAB</kbd> completion in message-mode to headers. It’s very fast, so feels more convenient than Thunderbird. </p> <dl class="org-dl"> <dt> Making it better </dt> <dd> I wish I could get org-contacts to provide tab completion in my subtrees (see below). I would need to access the completion function directly and somehow set the binding for <kbd>TAB</kbd> to that completion function. </dd> </dl> Adding Attachments with Drag & Drop After I make inline comments, I fill out a grading template and attach the paper to the resultant subtree (C-c C-a a PATH). This is OK, but sometimes it would nice to be able to drag and drom the files, so I am working on these functions. <dl class="org-dl"> <dt> Even better </dt> <dd> an even better solution be to add the attachments programmatically. The studnet papes follow a strict naming convention, so I should be able to crawl the directory and find the most recent paper with the student’s name in it… I’m worried it wil lbe too error prone though. </dd> </dl> <p> Anyway: unfortunately the following code doesn’t work right, so <b>don’t just cut and paste this code!). I *ought</b> to be able to bind the drag and drop action to a function – even several functions – and, if conditions are right, attach the dragged file to the current org header. John Kitchin describes this method <a href="">here</a>. But I do the following instead, which is also broken right now: </p> <p> Start by loading org-download, which downloads dragged images as attachments and inserts a link. (yay). THen a modification which fixes handling of file links allowing me to drag-n-drop files links onto org as attachments. Unfortunately, I can’t get org-attach to process the URI’s properly. Darn it. </p> <div class="org-src-container"> <pre class="src src-emacs-lisp"><span style="color: #707183;">(</span><span style="color: #a020f0;">require</span> '<span style="color: #008b8b;">org-download</span><span style="color: #707183;">)</span> (require ‘org-attach) ;; extending the dnd functionality ;; but doesn’t actually work… (defun mwp-org-file-link-dnd (uri action) “When in `org-mode’ and URI points to local file, add as attachment and also add a link. Otherwise, pass URI and Action back to dnd dispatch” (let ((img-regexp "\(png$\|jp[e]?g$\)") (newuri (replace-regexp-in-string “file:///" "/" uri))) (cond ((eq major-mode ‘org-mode) (message “Hi! newuri: %s “ (file-relative-name newuri)) (cond ((string-match img-regexp newuri) (insert "#+ATTR_ORG: :width 300\n”) (insert (concat "#+CAPTION: “ (read-input “Caption: “) "\n”)) (insert (format "[[%s]]" uri)) (org-display-inline-images t t)) (t (org-attach-new newuri) (insert (format "[[%s]]" uri)))) ) (t (let ((dnd-protocol-alist (rassq-delete-all ‘mwp-org-file-link-dnd (copy-alist dnd-protocol-alist)))) (dnd-handle-one-url nil action uri))) ))) ;; add a new function that DOESN’T open the attachment! (defun org-attach-new-dont-open (file) “Create a new attachment FILE for the current task. The attachment is created as an Emacs buffer." (interactive “sCreate attachment named: “) (when (and org-attach-file-list-property (not org-attach-inherited)) (org-entry-add-to-multivalued-property (point) org-attach-file-list-property file)) ) (defun mwp-org-file-link-enable () “Enable file drag and drop attachments." (unless (eq (cdr (assoc "^\(file\)://" dnd-protocol-alist)) ‘mwp-org-file-link-dnd) (setq dnd-protocol-alist `(("^\(file\)://" . mwp-org-file-link-dnd) ,@dnd-protocol-alist)))) (defun mwp-org-file-link-disable () “Enable file drag and drop attachments." (if (eq (cdr (assoc "^\(file\)://" dnd-protocol-alist)) ‘mwp-org-file-link-dnd) (rassq-delete-all ‘mwp-org-file-link-dnd dnd-protocol-alist) <span style="color: #707183;">))</span> (mwp-org-file-link-enable)

Exporting org-files to a temporary location

I have a private journal, which lives in an encrypted file in a Dropbox-backed-up directory. I use html export to examine the contents sometimes – there are some big old tables that are hard to read in org-mode – but I don’t want the html file to end up in Dropbox. So I just copied the definition of org-export-html-as-html and made trivial modifications. There’s probably a better way to do this. ;; export html to tmp dir (defun mwp-org-html-to-tmp (&optional async subtreep visible-only body-only ext-plist) "Export current buffer to a HTML file in the tmp directory. If narrowing is active in the current buffer, only export its narrowed part. If a region is active, export that region. A non-nil optional argument ASYNC means the process should happen asynchronously. The resulting file should be accessible through the `org-export-stack' interface. When optional argument SUBTREEP is non-nil, export the sub-tree at point, extracting information from the headline properties first. When optional argument VISIBLE-ONLY is non-nil, don’t export contents of hidden elements. When optional argument BODY-ONLY is non-nil, only write code between "<body>" and "</body>" tags. EXT-PLIST, when provided, is a property list with external parameters overriding Org default settings, but still inferior to file-local settings. Return output file’s name." (interactive) (let* ((extension (concat "." (or (plist-get ext-plist :html-extension) org-html-extension “html”))) ;; this is the code I’ve changed from the original function. (file (org-export-output-file-name extension subtreep "/home/matt/tmp/")) (org-export-coding-system org-html-coding-system)) (org-export-to-file 'html file async subtreep visible-only body-only ext-plist) (org-open-file file))) (org-defkey org-mode-map (kbd “C-c 0”) ‘mwp-org-html-to-tmp)

Creating and Publishing Presentations with Org-reveal

For several years, I’ve been using Org-mode to compose slides for my lectures. This method is great, because I get to work in plain-text and focus on the content of my lectures rather than animations; but it’s meant that when I want to share my presentations with others, there’s a certain amount of work involved as I move from a local copy on my computer to a web-based version. (This has largely been an issue because I sometimes compose my lectures sitting in a café with lousy Internet, and I sometimes give my lectures in a horrible classroom at U of T with terrible Internet reception.) I’ve now largely solved this problem, though there is hopefully an improvement coming down the pipe which will make it even easier. Org-mode has the capacity to export to a number of slide-like formats, including the LaTeX-based Beamer format, which also makes good PDF presentations, a couple of Emacs-based presentation tools, and a number of HTML5 formats. Since I teach about the web all the time, the HTML5 formats have always been the most appealing to me. Org-Reveal Setup I have used and still very much like deck.js (exporter here), but have recently switched to org-reveal, which I really like a lot. It’s not part of the official org distribution, so installation and setup are a little more involved, but not difficult. I just cloned the org-reveal and reveal.js repositories: <div class="org-src-container"> <pre class="src src-sh"><span style="color: #483d8b;">cd</span> ~/src git clone git clone <p> and put this in my <code>emacs-init.el</code>: </p> <div class="org-src-container"> <pre class="src src-emacs-lisp"><span style="color: #b22222;">;; </span><span style="color: #b22222;">org-reveal</span> (add-to-list ‘load-path "~/src/org-reveal") (require ‘ox-reveal) ;; set local root (setq org-reveal-root “file:///home/matt/src/reveal.js”) <p> That’s all that’s needed to get export working! I find it’s really fast to prepare lectures. </p> Publishing That’s great for giving lectures, and is all I really need at 9:55 when I’m trying to type my lecture and walk to class at the same time. But after lecture I want to put my slides somewhere my students can see them. Even if I wanted to, it would be impossible for me to post to Blackboard, which turns these files into garbage. What I want to do is publish them to the web; but I need to make sure that all the JS and CSS links are pointing to the web-based libraries and not my local copies, which of course no one but me can see. To do this I had to make one small change to org-reveal.el, which I have submitted as a pull request. This creates a new variable, org-reveal-extra-css, which I can refer to in my own functions. <p> Then I use org-mode’s fantastic built-in <a href="">Publishing functions</a> to push my slides to a public website. Publishing allows you to perform an export on many files, and customize the output in powerful ways that are mostly beyond me, actually. Still, I have a setup that I like a lot. </p> <p> First, my <code>org-publish-project-alist</code>, which defines the publishing targets. Note especially the top part, which defines “meta-projects”: for instance, I can publish all the slides and source files for all my classes with one command, <code>M-x org-publish-projects [RET] courses</code>. </p> <div class="org-src-container"> <pre class="src src-emacs-lisp">(<span style="color: #a020f0;">setq</span> org-publish-project-alist '( (<span style="color: #8b2252;">"courses"</span> <span style="color: #483d8b;">:components</span> (<span style="color: #8b2252;">"dh"</span> <span style="color: #8b2252;">"rlg231"</span>)) (<span style="color: #8b2252;">"rlg231"</span> <span style="color: #483d8b;">:components</span> (<span style="color: #8b2252;">"rlg231-lecture-slides"</span> <span style="color: #8b2252;">"rlg231-lecture-source"</span>)) (<span style="color: #8b2252;">"dh"</span> <span style="color: #483d8b;">:components</span> (<span style="color: #8b2252;">"digital-history-lecture-slides"</span> <span style="color: #8b2252;">"digital-history-lecture-source"</span>)) (<span style="color: #8b2252;">"rlg231-lecture-slides"</span> <span style="color: #483d8b;">:base-directory</span> <span style="color: #8b2252;">"~/RLG231/Lectures/"</span> <span style="color: #483d8b;">:base-extension</span> <span style="color: #8b2252;">"org"</span> <span style="color: #483d8b;">:publishing-directory</span> <span style="color: #8b2252;">"/ssh:matt@shimano:/var/www/sandbox/RLG231/Lectures/Slides"</span> <span style="color: #483d8b;">:recursive</span> t <span style="color: #483d8b;">:publishing-function</span> mwp-org-reveal-publish-to-html <span style="color: #483d8b;">:preparation-function</span> nil <span style="color: #483d8b;">:completion-function</span> nil <span style="color: #483d8b;">:headline-levels</span> 4 <span style="color: #b22222;">; </span><span style="color: #b22222;">Just the default for this project.</span> <span style="color: #483d8b;">:exclude</span> <span style="color: #8b2252;">""</span> <span style="color: #483d8b;">:exclude-tags</span> note noexport <span style="color: #483d8b;">:auto-preamble</span> t) (<span style="color: #8b2252;">"rlg231-lecture-source"</span> <span style="color: #483d8b;">:base-directory</span> <span style="color: #8b2252;">"~/RLG231/Lectures/"</span> <span style="color: #483d8b;">:base-extension</span> <span style="color: #8b2252;">"org"</span> <span style="color: #483d8b;">:publishing-directory</span> <span style="color: #8b2252;">"/ssh:matt@shimano:/var/www/sandbox/RLG231/Lectures/Source"</span> <span style="color: #483d8b;">:recursive</span> t <span style="color: #483d8b;">:publishing-function</span> org-org-publish-to-org <span style="color: #483d8b;">:preparation-function</span> nil <span style="color: #483d8b;">:completion-function</span> nil <span style="color: #483d8b;">:headline-levels</span> 4 <span style="color: #b22222;">; </span><span style="color: #b22222;">Just the default for this project.</span> <span style="color: #483d8b;">:exclude</span> <span style="color: #8b2252;">""</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:exclude ""</span> <span style="color: #483d8b;">:exclude-tags</span> note noexport <span style="color: #483d8b;">:auto-preamble</span> t) (<span style="color: #8b2252;">"digital-history-lecture-source"</span> <span style="color: #483d8b;">:base-directory</span> <span style="color: #8b2252;">"~/DH/Lectures"</span> <span style="color: #483d8b;">:base-extension</span> <span style="color: #8b2252;">"org"</span> <span style="color: #483d8b;">:publishing-directory</span> <span style="color: #8b2252;">"/ssh:matt@shimano:/var/www/sandbox/DigitalHistory/Lectures/Source"</span> <span style="color: #483d8b;">:recursive</span> t <span style="color: #483d8b;">:publishing-function</span> org-org-publish-to-org <span style="color: #483d8b;">:preparation-function</span> <span style="color: #483d8b;">:completion-function</span> <span style="color: #483d8b;">:headline-levels</span> 4 <span style="color: #b22222;">; </span><span style="color: #b22222;">Just the default for this project.</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:exclude ""</span> <span style="color: #483d8b;">:exclude</span> <span style="color: #8b2252;">""</span> <span style="color: #483d8b;">:exclude-tags</span> note noexport <span style="color: #483d8b;">:auto-preamble</span> t) (<span style="color: #8b2252;">"digital-history-lecture-slides"</span> <span style="color: #483d8b;">:base-directory</span> <span style="color: #8b2252;">"~/DH/Lectures"</span> <span style="color: #483d8b;">:base-extension</span> <span style="color: #8b2252;">"org"</span> <span style="color: #483d8b;">:publishing-directory</span> <span style="color: #8b2252;">"/ssh:matt@shimano:/var/www/sandbox/DigitalHistory/Lectures/Slides"</span> <span style="color: #483d8b;">:recursive</span> t <span style="color: #483d8b;">:publishing-function</span> mwp-org-reveal-publish-to-html <span style="color: #483d8b;">:preparation-function</span> <span style="color: #483d8b;">:completion-function</span> <span style="color: #483d8b;">:headline-levels</span> 4 <span style="color: #b22222;">; </span><span style="color: #b22222;">Just the default for this project.</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:exclude ""</span> <span style="color: #483d8b;">:exclude</span> <span style="color: #8b2252;">""</span> <span style="color: #483d8b;">:exclude-tags</span> note noexport <span style="color: #483d8b;">:auto-preamble</span> t) <span style="color: #b22222;">;; </span><span style="color: #b22222;">("newone-lecture-slides"</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:base-directory "~/NewOne/Lectures/"</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:base-extension "org"</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:publishing-directory "/ssh:matt@shimano:/var/www/sandbox/NewOne/Lectures"</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:recursive t</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:publishing-function org-deck-publish-to-html</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:headline-levels 4 ; Just the default for this project.</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:exclude-tags note noexport</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:auto-preamble t)</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">("newone-lecture-notes"</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:base-directory "~/NewOne/Lectures/"</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:base-extension "org"</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:publishing-directory "/ssh:matt@shimano:/var/www/sandbox/NewOne/Lectures-with-notes"</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:recursive t</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:publishing-function org-html-publish-to-html</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:headline-levels 4 ; Just the default for this project.</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:exclude-tags noexport</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:auto-preamble t)</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">("newone-images"</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:base-directory "~/NewOne/Images/"</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:base-extension "jpg\\|gif\\|png"</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:publishing-directory "/ssh:matt@shimano:/var/www/sandbox/NewOne/Images"</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:publishing-function org-publish-attachment)</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">("newone" :components ("newone-lecture-slides" "newone-lecture-notes" "newone-images") )</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">("presentations"</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:base-directory "~/Dropbox/Work/Talks/"</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:base-extension "org"</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:publishing-directory "/ssh:matt@shimano:/var/www/sandbox/Presentations"</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:headline-levels 4 ; just the default for this project</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:exclude-tags noexport</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:auto-preamble t</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">:publishing-function mwp-org-deck-publish-to-html</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">;; :completion-function mwp-update-published-paths</span> <span style="color: #b22222;">;; </span><span style="color: #b22222;">)</span> )) <p> Notice the publishing function, which is set to <code>mwp-org-deck-publish-to-html</code>. This is a simple function that resets the base url and <code>extra-css</code> values to web-based ones before publication, so that the presentations work when online. Notice I’ve also reset the <code>deck.js</code> base url, in case I ever decide to change back to deck. </p> <div class="org-src-container"> <pre class="src src-emacs-lisp">(<span style="color: #a020f0;">defun</span> <span style="color: #0000ff;">mwp-org-reveal-publish-to-html</span> (plist filename pub-dir) “Publish an org file to reveal.js HTML Presentation. FILENAME is the filename of the Org file to be published. PLIST is the property list for the given project. PUB-DIR is the publishing directory. Returns output file name." (let ((org-deck-base-url “") (org-reveal-root “") (org-reveal-extra-css “")) (org-publish-org-to 'reveal filename <span style="color: #8b2252;">".html"</span> plist pub-dir)) ) <p> And that’s it, magic! </p> Still to do I like this a lot, but there are a couple of pieces I’d still like to implement. <dl class="org-dl"> <dt> Fix all local file URL’s </dt> <dd> I’d like to write a function to take a final pass through all the links and change <code>file:///</code> links to <b>HTML relative links</b>. That will take some work though. </dd> <dt> Export as standalone </dt> <dd> There is work underway to allow presentations to be generated as stand-alone files that can be, e.g, sent by email. I like this idea a lot. <a href="">See this Github issue</a>. </dd> <dt> Standardize notes, fragments </dt> <dd> Every time I switch from one presentation framework to another, I have to learn a whole different syntax for things like fragments (bits of content that don’t appear on the slide immediately, but are instead stepped through) and speaker notes (that don’t appear on the slide that your viewers see, but are only visible to you in some kind of preview mode). It would be great if the various slide modes could work towards a common syntax for these things. If I have time, energy, and skills, I would like to help develop this a little. </dd> </dl> See my slides If you want to see some examples of the end product, here is a link to my Digital History lecture archive (still being built!). Many of my course materials are also online at Github.