gcv.appengine-magic

https://github.com/gcv/appengine-magic.git

git clone 'https://github.com/gcv/appengine-magic.git'

(ql:quickload :gcv.appengine-magic)
★349

The appengine-magic library attempts to abstract away the infrastructural nuts and bolts of writing a Clojure application for the Google App Engine platform.

The development environment of Google App Engine for Java expects pre-compiled classes, and generally does not fit well with Clojure's interactive development model. appengine-magic attempts to make REPL-based development of App Engine applications as natural as any other Clojure program.

  1. Programs using appengine-magic just need to include appengine-magic as a Leiningen dev-dependency.
  2. appengine-magic takes a Ring handler and makes it available as a servlet for App Engine deployment.
  3. appengine-magic is also a Leiningen plugin, and adds several tasks which simplify preparing for App Engine deployment.

Using appengine-magic still requires familiarity with Google App Engine. This README file tries to describe everything you need to know to use App Engine with Clojure, but does not explain the details of App Engine semantics. Please refer to Google's official documentation for details.

Please read the project's HISTORY file to learn what changed in recent releases.

Current Status

The code on this branch adds experimental support for App Engine SDK 1.7.4 and Leiningen 2.0. A stable older version is available at the v0.5.0 tag.

TODO for a stable 0.5.1 release

Dependencies

Overview

To use appengine-magic effectively, you need the following:

  1. The appengine-magic jar available on the project classpath.
  2. A Ring handler for your main application. You may use any Ring-compatible framework to make it. If your application does not yet have a core.clj file, then the lein appengine-new task creates one for you with a simple “hello world” Ring handler.
  3. A var defined by passing the Ring handler to the appengine-magic.core/def-appengine-app macro. This makes the application available both to interactive REPL development, and to App Engine itself.
  4. An entry point servlet. REPL development does not use it, but the standard App Engine SDK dev_appserver.sh mode and production deployment both do. This servlet must be AOT-compiled into a class file. This servlet defaults to the name app_servlet.clj, and the lein appengine-new task creates one for your project. The servlet must refer to the var defined by def-appengine-app.
  5. Web application resources. This primarily includes web application descriptors. lein appengine-new generates those and places them in the war/WEB-INF/ directory. You should also place all static files that your application uses in war/.

Here is a sample core.clj, using Compojure (other Ring-compatible frameworks, such as Moustache, also work):

(ns simple-example.core
  (:use compojure.core)
  (:require [appengine-magic.core :as ae]))

(defroutes simple-example-app-handler
  (GET "/" req
       {:status 200
        :headers {"Content-Type" "text/plain"}
        :body "Hello, world!"})
  (GET "/hello/:name" [name]
       {:status 200
        :headers {"Content-Type" "text/plain"}
        :body (format "Hello, %s!" name)})
  (ANY "*" _
       {:status 200
        :headers {"Content-Type" "text/plain"}
        :body "not found"}))

(ae/def-appengine-app simple-example-app #'simple-example-app-handler)

If you wish to emit HTML or XML from your application, you should use a specialized Clojure server-side templating library, e.g., Enlive or Hiccup. None of the appengine-magic examples rely on these libraries.

Getting Started

Project setup

You need a copy of the Google App Engine SDK installed somewhere. appengine-magic cannot replace its dev_appserver.sh and appcfg.sh functionality.

  1. lein new
  2. Optional: rm src/<project-name>/core.clj to clean out the default core.clj file created by Leiningen. You need to do this so that appengine-magic can create a default file which correctly invokes the def-appengine-app macro.
  3. Edit project.clj: add [appengine-magic "0.5.1-SNAPSHOT"] to both your :dependencies and :plugins.
  4. lein deps. This fetches appengine-magic, and makes its Leiningen plugin tasks available. If you already have the App Engine SDK installed locally, and do not wish to wait for Maven to download it again as a dependency, you may optionally run the provided install-artifacts.sh script first.
  5. lein appengine-new. This sets up four files for your project: core.clj (which has a sample Ring handler and uses the def-appengine-app macro), app_servlet.clj (the entry point for the application), war/WEB-INF/web.xml (a servlet descriptor), and war/WEB-INF/appengine-web.xml (an App Engine application descriptor). These files should contain reasonable starting defaults for your application.

With regard to AOT-compilation, if your project needs it, then you must include <project>.app_servlet in Leiningen's :aot directive. Otherwise, omit the :aot directive altogether. The lein appengine-prepare task will take care of AOT-compiling the entry point servlet and cleaning up afterwards.

The default .gitignore file produced by Leiningen works well with the resulting project, but do take a careful look at it. In particular, you should avoid checking in war/WEB-INF/lib/ or war/WEB-INF/classes/: let Leiningen take care of managing those directories.

Development process

Launch lein swank or lein repl, whichever you normally use. Once you have a working REPL, compile your application's core.clj (or whatever other entry point file you use).

The key construct provided by appengine-magic is the appengine-magic.core/def-appengine-app macro. It takes a Ring handler and defines a new <project-name>-app var. If you want to rename this var, remember to update app_servlet.clj. That's it: you may now write your application using any framework which produces a Ring-compatible handler. Then, just pass the resulting Ring handler to def-appengine-app.

To test your work interactively, you can control a Jetty instance from the REPL using appengine-magic.core/start and appengine-magic.core/stop. In addition, a convenience function, appengine-magic.core/serve, will either start or restart a running instance. Examples (assuming you are in your application's core namespace and your application is named foo):

(require '[appengine-magic.core :as ae])

;; recommended: use this to start or restart an app
(ae/serve foo-app)

;; or use these lower-level functions
(ae/start foo-app)
(ae/stop)
(ae/start foo-app :port 8095)
(ae/stop)

Recompiling the functions which make up your Ring handler should produce instantaneous results.

If you use SLIME, then the swank.core/break function works even inside a Ring handler.

Testing with dev_appserver.sh

  1. lein appengine-prepare. This AOT-compiles the entry point servlet, makes a jar of your application, and copies it, along with all your library dependencies, to your application's war/WEB-INF/lib/ directories.
  2. Run dev_appserver.sh with a path to your application's war/ directory.

Static files

Just put all static files into your application's war/ directory. If you put a file called index.html there, it will become a default welcome file.

Classpath resources

Put all classpath resources you expect to need at runtime in resources/. You can then access them using the appengine-magic.core/open-resource-stream, which returns a java.io.BufferedInputStream instance.

You may also use appengine-magic.core/resource-url to find a classpath resource's internal URL. This URL will not be externally visible (it will not be an HTTP URL), but you may use it to refer to classpath resources from within the application's code.

Do not use direct methods like java.io.File or ClassLoader/getSystemClassLoader to access classpath resources; they do not work consistently across all App Engine environments.

Deployment to App Engine

  1. First of all, be careful. You must manually maintain the version field in appengine-web.xml and you should understand its implications. Refer to Google App Engine documentation for more information.
  2. lein appengine-prepare prepares the war/ directory with the latest classes and libraries for deployment.
  3. When you are ready to deploy, just run appcfg.sh update with a path to your application's war/ directory.

Checking the runtime environment

Automatic testing code

The clojure.test system works well for testing appengine-magic applications, but all tests must bootstrap App Engine services in order to run. The appengine-magic.testing namespace provides several functions usable as clojure.test fixtures to help you do so. The easiest way to get started is:

(use 'clojure.test)
(require '[appengine-magic.testing :as ae-testing])

(use-fixtures :each (ae-testing/local-services :all))

Then, write deftest forms normally; you can use App Engine services just as you would in application code.

File uploads and multipart forms

A Ring application requires the use of middleware to convert the request body into something useful in the request map. Ring comes with ring.middleware.multipart-params/wrap-multipart-params which does this; unfortunately, this middleware uses classes restricted in App Engine. To deal with this, appengine-magic has its own middleware.

appengine-magic.multipart-params/wrap-multipart-params works just like the Ring equivalent, except file upload parameters become maps with a :bytes key (instead of :tempfile). This key contains a byte array with the upload data.

A full Compojure example (includes features from the Datastore service):

(use 'compojure.core
     '[appengine-magic.multipart-params :only [wrap-multipart-params]])

(require '[appengine-magic.core :as ae]
         '[appengine-magic.services.datastore :as ds])

(ds/defentity Image [^{:tag :key} name, content-type, data])

(defroutes upload-images-demo-app-handler
  ;; HTML upload form
  (GET "/upload" _
       {:status 200
        :headers {"Content-Type" "text/html"}
        :body (str "<html><body>"
                   "<form action=\"/done\" "
                   "method=\"post\" enctype=\"multipart/form-data\">"
                   "<input type=\"file\" name=\"file-upload\">"
                   "<input type=\"submit\" value=\"Submit\">"
                   "</form>"
                   "</body></html>")})
  ;; handles the uploaded data
  (POST "/done" _
        (wrap-multipart-params
         (fn [req]
           (let [img (get (:params req) "file-upload")
                 img-entity (Image. (:filename img)
                                    (:content-type img)
                                    (ds/as-blob (:bytes img)))]
             (ds/save! img-entity)
             {:status 200
              :headers {"Content-Type" "text/plain"}
              :body (with-out-str
                      (println (:params req)))}))))
  ;; hit this route to retrieve an uploaded file
  (GET ["/img/:name", :name #".*"] [name]
       (let [img (ds/retrieve Image name)]
         (if (nil? img)
             {:status 404}
             {:status 200
              :headers {"Content-Type" (:content-type img)}
              :body (.getBytes (:data img))}))))

(ae/def-appengine-app upload-images-demo-app #'upload-images-demo-app-handler)

Please note that you do not need to use this middleware with the Blobstore service. App Engine takes care decoding the upload in its internal handlers, and the upload callbacks do not contain multipart data.

Managing multiple environments

Most web applications use several environments internally: production, plus various staging and development installations. App Engine supports multiple versions in its appengine-web.xml file, but does nothing to help deal with installing to different full environments. Since different versions of App Engine applications share the same blobstore and datastore, distinguishing between production and staging using only versions is dangerous.

appengine-magic has a mechanism to help deal with multiple environments. The Leiningen appengine-update task replaces the use of appcfg.sh update, and a new entry in project.clj manages applications and versions.

  1. Rename your WEB-INF/application-web.xml file to WEB-INF/application-web.xml.tmpl. For safety reasons, appengine-update will not run if a normal application-web.xml exists. For clarity, you should blank out the contents of the <application> and <version> tags of the template file (but leave the tags in place).

  2. Add a new entry to project.clj: :appengine-app-versions. This entry is a map from application name to application version. Example:

     :appengine-app-versions {"myapp-production" "2010-11-25 11:15"
                          "myapp-staging"    "2010-11-27 22:05"
                          "myapp-dev1"       "2830"
                          "myapp-dev2"       "2893"}

    The myapp- key strings correspond to App Engine applications, registered and managed through the App Engine console. The value strings are the versions appengine-update will install if invoked on that application.

  3. Add a new entry to project.clj: :appengine-sdk. The App Engine SDK location is necessary to execute the actual production deployment. This value can be just a string, representing a path. Alternatively, for teams whose members keep the App Engine SDK in different locations, this value can be a map from username to path string. Examples:

     :appengine-sdk "/opt/appengine-java-sdk"
 :appengine-sdk {"alice"   "/opt/appengine-java-sdk"
                 "bob"     "/Users/bob/lib/appengine-java-sdk"
                 "charlie" "/home/charlie/appengine/sdk/current"}

    If the APPENGINE_HOME environment variable is set, its value will be used if no :appengine-sdk entry is found in the project.clj file.

  4. Run lein appengine-update <application>, where the argument is an application name from the :appengine-app-versions map.

If you use this mechanism, be aware that dev_appserver.sh will no longer work (since your project no longer defines a simple appengine-web.xml file). To run that process, use lein appengine-dev-appserver <application>.

You may also force a specific version string if you pass it as an optional argument to appengine-update: lein appengine-update <application> <version>.

App Engine Services

appengine-magic provides convenience wrappers for using App Engine services from Clojure. Most of these API calls will work when invoked from the REPL, but only if an application is running — that is, it was launched using appengine-magic.core/start.

User service

The appengine-magic.services.user namespace provides the following functions for handling users.

Memcache service

The appengine-magic.services.memcache namespace provides the following functions for the App Engine memcache. See App Engine documentation for detailed explanations of the underlying Java API.

Datastore

The appengine-magic.services.datastore namespace provides a fairly complete interface for the App Engine datastore.

A few simple examples:

(require '[appengine-magic.services.datastore :as ds])

(ds/defentity Author [^{:tag :key} name, birthday])
(ds/defentity Book [^{:tag :key} isbn, title, author])

;; Writes three authors to the datastore.
(let [will (Author. "Shakespeare, William" nil)
      geoff (Author. "Chaucer, Geoffrey" "1343")
      oscar (Author. "Wilde, Oscar" "1854-10-16")]
  ;; First, just write Will, without a birthday.
  (ds/save! will)
  ;; Now overwrite Will with an entity containing a birthday, and also
  ;; write the other two authors.
  (ds/save! [(assoc will :birthday "1564"), geoff, oscar]))

;; Retrieves two authors and writes book entites.
(let [will (first (ds/query :kind Author :filter (= :name "Shakespeare, William")))
      geoff (first (ds/query :kind Author :filter [(= :name "Chaucer, Geoffrey")
                                                   (= :birthday "1343")]))]
  (ds/save! (Book. "0393925870" "The Canterbury Tales" geoff))
  (ds/save! (Book. "143851557X" "Troilus and Criseyde" geoff))
  (ds/save! (Book. "0393039854" "The First Folio" will)))

;; Retrieves all Chaucer books in the datastore, sorting by descending title and
;; then by ISBN.
(let [geoff (ds/retrieve Author "Chaucer, Geoffrey")]
  (ds/query :kind Book
            :filter (= :author geoff)
            :sort [[title :dsc] :isbn]))

;; Deletes all books by Chaucer.
(let [geoff (ds/retrieve Author "Chaucer, Geoffrey")]
  (ds/delete! (ds/query :kind Book :filter (= :author geoff))))

The next example (which uses Compojure) demonstrates the use of entity groups and transactions.

(use '[clojure.pprint :only [pprint]]
     'compojure.core)
(require '[appengine-magic.core :as ae]
         '[appengine-magic.services.datastore :as ds])

(ds/defentity Parent [^{:tag :key} name, children])
(ds/defentity Child [^{:tag :key} name])

(defroutes entity-group-example-app-handler
  (GET  "/" [] {:headers {"Content-Type" "text/plain"} :body "started"})
  (POST "/new/:parent-name/:child-name" [parent-name child-name]
        (let [parent (or (ds/retrieve Parent parent-name)
                         ;; Note the use of ds/save! here. Unless an entity has
                         ;; been saved to the datastore, children cannot join
                         ;; the entity group.
                         (ds/save! (Parent. parent-name [])))
              ;; Note the use of ds/new* here: it is required so that a :parent
              ;; entity may be specified in the instantiation of a child entity.
              child (ds/new* Child [child-name] :parent parent)]
          ;; Updating the parent and the child together occurs in a transaction.
          (ds/with-transaction
            (ds/save! (assoc parent
                        :members (conj (:children parent) child-name)))
            (ds/save! child))
          {:headers {"Content-Type" "text/plain"}
           :body "done"}))
  (GET  "/parents" []
        (let [parents (ds/query :kind Parent)]
          {:headers {"Content-Type" "text/plain"}
           :body (str (with-out-str (pprint parents))
                      "\n"
                      (with-out-str (pprint (map ds/get-key-object parents))))}))
  (GET  "/children" []
        (let [children (ds/query :kind Child)]
          {:headers {"Content-Type" "text/plain"}
           :body (str (with-out-str (pprint children))
                      "\n"
                      (with-out-str (pprint (map ds/get-key-object children))))}))
  (ANY  "*" [] {:status 404 :body "not found" :headers {"Content-Type" "text/plain"}}))

The Clojure interface to the Datastore has an additional feature: any entity field may be marked with the ^:clj metadata tag:

(ds/defentity TestEntity [^{:tag :key} test-id, ^:clj some-table])

The values of fields marked with the ^:clj tag will go into the datastore as strings produced by Clojure's prn-str function, and they will be retrieved as Clojure objects read by read-string. In other words, ^:clj fields will be serialized and retrieved using Clojure's reader. This is quite helpful for dealing with types which the datastore does not support: specifically maps (not even java.util.HashMap works) and sets (not even java.util.HashSet works). Keep in mind, however, that these fields are stored as instances of com.google.appengine.api.datastore.Text, which the datastore does not index.

Blobstore

The appengine-magic.services.blobstore namespace helps with the App Engine Blobstore service, designed for hosting large files. Note that the production App Engine only enables the Blobstore service for applications with billing enabled.

Using the Blobstore generally requires three components: an upload session, an HTTP multipart/form-data file upload (usually initiated through an HTML form), and an upload callback.

  1. Your application must first initiate an upload session; this gives it a URL to use for the corresponding HTTP POST request.
  2. Your application must provide a proper upload form, with the action pointing to the URL of the upload session, the method set to post, and enctype set to multipart/form-data; each uploaded file must have a name attribute.
  3. Your application must provide an upload callback URL. App Engine will make an HTTP POST request to that URL once the file upload completes. This callback's request will contain information about the uploaded files. The callback should save this data in some way that makes sense for the application. The callback implementation must end with an invocation of the callback-complete function. Do not attempt to return a Ring response map from an upload handler.
  4. A Ring handler which serves up a blob must end with an invocation of the serve function. Do not attempt to return a Ring response map from a blob-serving handler.

NB: In the REPL environment and in dev_appserver.sh, using the Blobstore writes entities into the datastore: __BlobInfo__ and __BlobUploadSession__. This does not happen in the production environment.

This is confusing, but a Compojure example will help.

(use 'compojure.core)

(require '[appengine-magic.core :as ae]
         '[appengine-magic.services.datastore :as ds]
         '[appengine-magic.services.blobstore :as blobs])

(ds/defentity UploadedFile [^{:tag :key} blob-key])

(defroutes upload-demo-app-handler
  ;; HTML upload form; note the upload-url call
  (GET "/upload" _
       {:status 200
        :headers {"Content-Type" "text/html"}
        :body (str "<html><body>"
                   "<form action=\""
                   (blobs/upload-url "/done")
                   "\" method=\"post\" enctype=\"multipart/form-data\">"
                   "<input type=\"file\" name=\"file1\">"
                   "<input type=\"file\" name=\"file2\">"
                   "<input type=\"file\" name=\"file3\">"
                   "<input type=\"submit\" value=\"Submit\">"
                   "</form>"
                   "</body></html>")})
  ;; success callback
  (POST "/done" req
       (let [blob-map (blobs/uploaded-blobs req)]
         (ds/save! [(UploadedFile. (.getKeyString (blob-map "file1")))
                    (UploadedFile. (.getKeyString (blob-map "file2")))
                    (UploadedFile. (.getKeyString (blob-map "file3")))])
         (blobs/callback-complete req "/list")))
  ;; a list of all uploaded files with links
  (GET "/list" _
       {:status 200
        :headers {"Content-Type" "text/html"}
        :body (apply str `["<html><body>"
                           ~@(map #(format " <a href=\"/serve/%s\">file</a>"
                                           (:blob-key %))
                                  (ds/query :kind UploadedFile))
                           "</body></html>"])})
  ;; serves the given blob by key
  (GET "/serve/:blob-key" {{:strs [blob-key]} :params :as req}
       (blobs/serve req blob-key)))

(ae/def-appengine-app upload-demo-app #'upload-demo-app-handler)

Note that the Blobstore API primarily allows for browser-driven file uploads. appengine-magic includes a hack which allows an application to upload a blob without a browser.

Mail service

The appengine-magic.services.mail namespace provides helper functions for sending and receiving mail in an App Engine application.

To send an mail message, construct it using make-message and make-attachment functions, and send it using the send function.

To receive incoming mail, first read and understand the relevant section in (Google's official documentation)[http://code.google.com/appengine/docs/java/mail/receiving.html]. You need to modify your application's appengine-web.xml, and you should add a security constraint for /_ah/mail/* URLs in your web.xml. In your application add a Ring handler for POST methods for URLs which begin with /_ah/mail.

NB: With Compojure, the only route which seems to work in the production App Engine for handling mail is /_ah/mail/*.

(use 'compojure.core)

(require '[appengine-magic.core :as ae]
         '[appengine-magic.services.mail :as mail])

(defroutes mail-demo-app-handler
  ;; sending
  (GET "/mail" _
       (let [att1 (mail/make-attachment "hello.txt" (.getBytes "hello world"))
             att2 (mail/make-attachment "jk.txt" (.getBytes "just kidding"))
             msg (mail/make-message :from "one@example.com"
                                    :to "two@example.com"
                                    :cc ["three@example.com" "four@example.com"]
                                    :subject "Test message."
                                    :text-body "Sent from appengine-magic."
                                    :attachments [att1 att2])]
         (mail/send msg)
         {:status 200
          :headers {"Content-Type" "text/plain"}
          :body "sent"}))
  ;; receiving
  (POST "/_ah/mail/*" req
       (let [msg (mail/parse-message req)]
         ;; use the resulting MailMessage object
         {:status 200})))

(ae/def-appengine-app mail-demo-app #'mail-demo-app-handler)

Task Queues service

The appengine-magic.services.task-queues namespace has helper functions for using task queues. As always, read Google's documentation on task queues, in particular the sections on configuring queue.xml, and on securing task URLs in web.xml. In addition, the section on scheduled tasks (cron.xml) is useful.

Use the add! function to add a new task to a queue, and provide a callback URL which implements the actual work performed by the task.

URL Fetch service

appengine-magic.services.url-fetch lets App Engine applications send arbitrary HTTP requests to external services.

Images service

With appengine-magic.services.images, an application can (1) apply simple transformations to images, either in the blobstore or saved in byte arrays, and (2) access blobstore images through a CDN, with limited resizing capability.

Channel service

App Engine has an implementation of server push through its Channel service (appengine-magic.services.channel). Using it requires a combination of client-side JavaScript event callbacks, and channel management on the server.

Conceptually, the server maintains one or more channels associated with a client ID (this is a small number; it is probably safest to assume only one channel per ID). The server opens a channel, which generates a channel token. This token must be passed to the connecting client; the client then uses the token to receive messages from the server.

NB: The current version of the Channel service does not help with channel bookkeeping. It probably cleans up idle channels internally, but does not inform the application of this. The application is responsible for keeping track of active channels.

The client needs to load the JavaScript code at /_ah/channel/jsapi:

<script src="/_ah/channel/jsapi" type="text/javascript"></script>

Once this library loads, the client must initiate a request in which the server can return the channel ID. Once this is done, the rest of the client API looks like this:

// read this from a normal server response
var channel_token = ...;

// open a "socket" to the server
var channel = new goog.appengine.Channel(channel_token);
var socket = channel.open();

// implement these callbacks to take action when an event occurs
socket.onopen = function(evt) { var data = evt.data; ... };
socket.onmessage = function(evt) { var data = evt.data; ... };
socket.onerror = function(evt) { var data = evt.data; ... };
socket.onclose = function(evt) { var data = evt.data; ... };

NB: The development implementations of the Channel service just poll the server for updates, and merely emulate server push. If you watch a browser request console, you'll see the polling requests.

Limitations

Incomplete features

The following Google services are not yet tested in the REPL environment:

They may still work, but appengine-magic does not provide convenient Clojure interfaces for them, and may lack mappings for any necessary supporting URLs.

Warning

Google App Engine maintains a whitelist of permitted classes in Java's standard library. Other classes will cause your application to fail to deploy. Examples include threads and sockets. If you use those in your application, it will not work. This means that you cannot use Clojure's agents or futures. In addition, if one of your dependencies uses those, your application will also not work. For example, clojure.java.io (and its fore-runner, duck-streams from clojure-contrib), uses java.net.Socket, a forbidden class.

Whenever you add a new dependency, no matter how innocuous, you should make sure your app still works. dev_appserver.sh is a good place to start, but you must also test in the main App Engine. The two do not always load classes the same way.

Contributors

Many thanks to:

License

appengine-magic is distributed under the MIT license.