Racket v5.2

Racket version 5.2 is now available from

http://racket-lang.org/

Release Highlights:

• DrRacket comes with an experimental, on-line check syntax tool, although this new tool is disabled default. See below for more information.
• The new db library offers a high-level, functional interface to popular relational database systems, including PostgreSQL, MySQL, and SQLite, as well as other systems via ODBC.
• A new XREPL collection provides convenient commands for a plain racket REPL. It is particularly convenient for people who prefer console-based work and alternative editors. See also the new chapter on command-line tools and other editors at the end of the Racket Guide.
• The plot collection has been reimplemented in Racket. It now offers PDF output, log axes, histograms, and more. Some code that uses plot will still work, and some will need light porting. The plot/compat module offers expedient backward compatibility.
• DrRacket uses more conventional key bindings: C-t creates a new tab, C-w closes the current one, and C-r runs the definitions. On Mac OS X, the Command key is used. See “Defining Custom Shortcuts” in the DrRacket manual for an example that uses the old key bindings.
• The new raco link command registers a directory as a collection, which allows the collection directory to reside outside the “collects” tree and without changing the PLTCOLLECTS environment variable.
• Typed Racket:
  • Typed Racket provides static performance debugging support to show which code gets optimized and point out code that does not. Use the “Performance Report” button in DrRacket.
  • More intuitive types in printouts in the REPL and in error messages. Use :query-result-type to explore types, or :print-type for a full printout.
  • Typed Racket now supports defining function with optional arguments using the same syntax as Racket.
• Redex now supports specifying (and testing and automatically typesetting) judgment forms including type systems and SOS-style operational semantics.
• Fixed several GUI problems, including problems on Ubuntu 11.10 (GTK+ 3) and 64-bit Mac OS X.
• Internal-definition expansion has changed to use let* semantics for sequences that contain no back references. This change removes a performance penalty for using internal definitions instead of let in common cases, and it only changes the meaning of programs that capture continuations in internal definitions. Internal definitions are now considered preferable in style to let.
• Support for begin-for-syntax has been generalized; modules may now define and export both value bindings and syntax bindings (macros) at phase 1 and higher.

  Due to a bug, phase 1 syntax (or higher) is not available in DrRacket's #lang-based REPL. A simple workaround is to disable debugging in DrRacket (see “no debugging” radio button in detailed language dialog).

Additional Items:

• The racket/gui library (and Slideshow) provides more support for multiple-screen displays.
• DrRacket remembers whether an opened file used LF or CRLF line endings, and will continue using the same. When creating a new file, a preference determines how it is saved.
• net/url can now follow HTTP redirections.
• The LNCS and JFP class files are no longer distributed with Racket. Instead, they are downloaded on demand.
• The Algol language implementation is now available as a plain language using #lang algol60.
• The Racket-to-C compiler (as accessed via raco ctool or mzc) has been removed; Racket's JIT has long provided better performance, and the FFI provides better access to C libraries.
• Contracts can be applied to exports with the new contract-out form within provide, instead of a separate provide/contract form. (The new contract-out form is implemented as a new kind of “provide pre-transformer”.)
• The date* structure type is an extension of date with nanosecond and time-zone-name fields.
• New looping constructs: for/sum and for/product.
• Direct calls to keyword-accepting functions are now optimized to eliminate the overhead of keywords. In addition, the compiler detects and logs warnings for keyword-argument mismatches.
• The libfit interface is available from plot/deprecated/fit, and will be removed in the near future.
• The Unix installer has been re-done, and it is now more robust.
• The built-in reader and printer support for Honu is removed. (This functionality is re-implemented in Racket.)

On-line Check Syntax:

DrRacket now provides an on-line version of the syntax check tool, which means that syntax checking runs automatically while you continue to edit a program. With this tool enabled, its annotations (e.g., binding arrows) and actions (e.g., the renaming refactoring and direct documentation links) are almost always available.

We have noticed that on-line syntax checking renders DrRacket unstable on occasion, perhaps because it relies on relatively new support for parallelism. Occurrences of the problem are rare, but they are not rare enough, which is why we have disabled the tool by default. At the same time, current users of the tool find it so valuable that we felt it should be included in the release. We expect to track down the remaining problems and enable the tool by default in near-future release.

To enable on-line syntax checking (for #lang-based programs only), click on the red dot in the bottom right of DrRacket's window. To turn it off, click there again.

On eval in dynamic languages generally and in Racket specifically

The eval function is at the heart of a dynamic language, and it strikes many newcomers as an amazingly powerful tool. At the same time, experienced programmers avoid eval, because unnecessary use creates trouble. It’s not easy to explain why eval should be avoided or when it”s appropriate to use eval, but I’ll take another stab at it here.

What is eval?

Consider the following “program” in English prose:

Assume that your favorite color is red. Now imagine a balloon that is your favorite color. Paint a canvas the same color as the balloon.

As English goes, that’s a fairly clear program with a fairly well-defined result. When I follow those instructions, at least, I will always produce a red canvas (assuming that I have a canvas and some red paint, but a potential lack of art supplies is not the point here).

I would come up with a red canvas even if I read the instructions when surrounded by people who speak only Chinese, obviously, since I’m the one reading the instructions. Furthermore, it would be straightforward to translate the program to Chinese, and then a person who reads Chinese would produce a red canvas.

A translator might even take the liberty of simplifying the program to just

Paint a canvas red.

The translation loses some of the poetry of the original, but the result is the same.

In Racket terms, the paragraph corresponds to a module. It can be compiled (i.e., translated) and optimized (i.e., simplified). A program can be made up of multiple modules that are written in different languages, but since each module can be reliably translated, they can all be compiled into some common language to run the program.

Here’s a different program:

Tell the person next to you “Assume that your favorite color is red.” Tell the person “Now, imagine a balloon that is your favorite color.” Tell the person “Paint canvas the same color as the balloon.”

Getting a red canvas back may be a little trickier in this case. If the person next to me speaks only Chinese, then my program may fail with a message-not-understood error.

If I want to translate the program to Chinese, then it’s not clear whether the parts in quotes should be translated. Maybe I mean for a person who can read Chinese but only sound out English to run the program when surrounded by English speakers, or maybe I mean for a Chinese person to run the program when surrounded by Chinese people. Either way, I have to be a lot more specific to a translator. For more complex programs, the instructions to the translator can become complex and fragile.

Finally, a translator probably won’t feel comfortable simplifying the program to

Tell the person next to you “Paint a canvas red.”

because there could be all sorts of environmental conditions that make the result different—such as people who are willing to paint but unwilling to accept assumptions about their favorite colors.

The paragraph with “tell the person...” is a program that uses eval. It can’t be compiled and optimized as well as the earlier paragraph, and the language context in which it is run may change the result. The quotes around sentences correspond to the quote in front of an expression passed to eval in Racket; there’s no particular reason that the language for eval will match the language of the program that has the quoted text. The issues become even more complex if you try to implement different parts of the program in different languages.

If the analogy to multiple spoken languages seems strange—maybe your language is Javascript, period—the problem of translation to another language is really a proxy for program understanding. There’s a direct connection to performance and optimization (i.e., translation to efficient machine code), but using eval also makes a program more difficult to understand for the same reasons that it makes the program more difficult to translate. For example, a reader of your program may not be able to tell whether “assume your favorite color is red” is just a rhetorical device to get to a red canvas or whether some new instructions will arrive that will ask for your favorite color.

When is eval Good?

The program with “tell the person next to you” above uses eval in a bad way. The task could just as well be performed by the person reading the instructions, instead of getting another nearby person involved.

Some other uses eval are both good and necessary. For example, consider the following program:

Ask the construction manager for instructions. Walk to the building site and convey those instructions to the construction crew.

This program uses eval when it conveys instructions to the construction crew, but no quoted forms appear in the program. The absence of quoted code is one sign that eval may be appropriate. Note that the program could work no matter what language the manager and crew speak, although there is an implicit (and sometimes non-trivial) assumption that the manager and crew speak the same language.

Here’s another example:

Go outside, and tell each member of the construction crew “take a lunch break, now.”

There’s a quoted program in this case, but it’s crucial to ask other people to run the quoted program, instead of just taking the lunch break yourself. That is, eval is really necessary. The implementor of this program takes on the burden of making sure that the instructions are in a suitable language, however, and may need to parameterize the quoted program by an explicit action to translate it to a language understood by the construction crew.

Here’s one more reasonable example:

Ask the construction manager for instructions. Follow them.

In this case, it’s the construction manager’s problem to give you instructions in a language that you understand.

Here’s a questionable example:

Decide how long to work before lunch, say N hours, and write a note to yourself to work N hours. Add to the note by telling yourself to take a lunch break afterward.

If you could really write that program without quotes, then it’s probably ok. The example is misleading, though, because languages don’t usually support

write a note to yourself to work N hours

You’d have to write instead

write a note to yourself that says “work” followed by the number N and then “hours”

and the quote marks are where the problem comes in. If you translate the program to Chinese, then you have to be careful to somehow translate “work” and “hours” to Chinese, too.

The point here is not that programs without quoted text are clearly good or that programs with quoted text are clearly bad. The real point is that a programmer has to be especially careful about passing around instructions and using quoted instructions. Using eval means accepting the burden of using instructions will make sense by the time they are delivered. That burdened is best avoided, which is why experienced programmers avoid eval, but some of the examples illustrate cases where the burden is not avoidable or where the actions enabled by eval make the burden worthwhile.

Using eval in Racket

In the context of Racket, the multiple-language analogy is relatively accurate, because Racket is about having many programming languages work together and allowing programmers to define ever better languages and language constructs. In Racket, it’s especially likely that a library written in one language is used in a context where another language is the default.

Newcomers to Racket sometimes stumble over the fact that

 #lang racket
 (define my-x 1)
 (eval '(+ my-x 2))

or even

 #lang racket
 (eval '(+ 1 2))

does not work at all, and yet if the program

 #lang racket
 (define my-x 1)

is loaded into a read-eval-print loop—for example, by clicking the “Run” button in DrRacket and then typing into the lower interactions panel—then

 (eval '(+ my-x 2))

works as expected.

DrRacket’s interactions window has to use eval in the sense that it reads an expression to evaluate and then passes it on to the interpreter for an answer. More generally, to make various pieces of the environment fit together, DrRacket sets eval globally to use the module’s language while evaluating expressions in the interactions window. In Racket terminology, DrRacket sets the current-namespace parameter to the module’s namespace when it initializes the interactions window. In contrast, while the module body is being evaluated, eval treats expressions as being in the language that is empty by default, which is why eval during the module evaluation produces a different result from eval during the interactions windows.

You may wonder why DrRacket doesn’t initialize the namespace of eval to be the module’s namespace from the start, so that in-module uses of eval and the interactions window behave the same. In a program that is implemented by multiple modules, which module’s language should be used? In particular, if the language it’s always the main module’s language, then a module may behave differently on its own than as part of a larger program. In the process of developing Racket and DrRacket, we’ve seen many such problems, and so Racket now arranges for the default language to be empty (which is different from any useful language) to help programmers remember that there’s a language issue to consider whenever eval is used.

The Racket Guide’s chapter 15 covers in more depth the issues and namespace tools of Racket for harnessing the power of eval:

http://docs.racket-lang.org/guide/reflection.html

Think of eval as a power tool. For some tasks, there’s no real substitute, and so we want eval around. At the same time, eval should be used with care. In dynamic languages generally, that means a reluctant and targeted use eval. In Racket specifically, it means knowing the namespace toolbox and being as explicit as possible about the intended context for dynamic evaluation.

Racket v5.1.3

Racket version 5.1.3 is now available from

  http://racket-lang.org/

This is a bugfix release, resolving the DrRacket issue with the contour view. In addition, two tex files with problematic licensing were removed.

Racket v5.1.2

Racket version 5.1.2 is now available from

  http://racket-lang.org/

• The download page includes 64-bit installers for Mac OS X, Windows, and two Debian flavors. Racket now supports OS X Lion.
• Racket now includes a new racket/place library to support parallelism, complementing racket/future. Racket's parallel build process is now based on places instead of multiple OS processes.
  Places support share-nothing parallelism and message-passing communication. Compared to futures, places are heavyweight, but they have a simpler performance model.
• The syntax-certificate system has been replaced by a syntax-taint system. Both certificates and taints were designed to protect otherwise inaccessible bindings from abuse when they appear in macro expansions. Taints are simpler and lighter, and the switch closes known holes in the certificate system. Macros that are not implemented with syntax-rules or define-syntax-rule, however, must explicitly use syntax-protect to protect their expansions from abuse.
• The net/url library supports HTTPS connections, but beware that by default all sites are accepted (equivalent to ignoring a browser's warnings about untrusted certificates).
• Error messages in the student languages use a simplified vocabulary and consistent phrasings. If you maintain curriculum material or teachpacks then please consider updating. See the “Error Message Composition Guidelines” section in the documentation for details.
• Typed Racket: almost all core Racket data structures and operations are now accessible in Typed Racket (most of this work is due to prolific contributor Eric Dobson). The performance of the typechecker has been significantly improved.
• The scriblib/bibtex library supports BibTeX-formatted citation databases in Scribble documents. BibTeX can be tricky to parse, so please report failed entries as bug reports.
• The for forms now support an #:unless clause, and a nonnegative integer can be used as a sequence. The new compose1 function creates single-valued composition functions. The racket/function library now provides identity, thunk, and thunk*.
• The license has been clarified: we now use LGPLv2.1 uniformly. (The license file used to specify LGPLv2, contrary to the download pages.)

RacketCon Recap

We held RacketCon here at Northeastern over the past weekend, and it was a great time!  Matthew Flatt consulted his magic 8-ball about the future of Racket (it involves Hawaiian shirts); Matthias Felleisen announced the upcoming Realm of Racket book; Robby Findler demoed DrRacket with online syntax checking; Ryan Culpepper taught us how to make Rackety libraries; Prabhakar Ragde explained how the University of Waterloo teaches Racket and Program by Design to 1500 students a year (and how to embed math into Scribble documents).

Danny Yoo has already blogged about his talk about his Whalesong Racket->JavaScript compiler. Hopefully other people will write more about some of the individual talks; I'll update this post to link to any that I see.

As promised, the talks were videotaped by Jeff Dlouhy of the NUACM; the videos are in post-production and should be up within a few weeks. We'll announce their availability on this blog.  Slides from the individual talks will be up on the RacketCon webpage soon.

Come to RacketCon!

Calling All Racketeers!

Join us at RacketCon, 23 & 24 July 2011.

http://con.racket-lang.org/

The schedule for RacketCon is now available, and includes presentations by key members of the Racket team, including Matthew Flatt on the future of Racket and Matthias Felleisen on the Program By Design curriculum, as well as tutorials on building web applications and accessing low-level libraries.

The meeting will take place at Northeastern University in Boston, MA. It will be an opportunity for developers, researchers, and educators using Racket, DrRacket, Program By Design, and related technologies to come together to share plans, ideas, and enthusiasm. To make it accessible, we will not charge any registration fees; we'll also serve you breakfast and lunch.

We're excited about RacketCon, and hope you'll join us there. To register for RacketCon, please send an email with your name and affiliation to racketcon@racket-lang.org.

Additional information, including registration, transportation and hotel information, can be found on the web page: http://con.racket-lang.org

Multi-file code coverage viewing tool

I'm very pleased to announce the availability of a multi-file code coverage viewer, written by Jonathan Walsh.
Torn between separating your test cases into another file and actually seeing the coverage? Well, go ahead and pull them apart, because the multi-file coverage tool displays coverage information for the files required by the present one, including both percentage covered (on a line-by-line basis) and optionally a list of uncovered lines (no more inching through your code, looking for the red highlighting.
Back End:
One reason I expect this tool to be long-term robust is that it makes absolutely no changes to the back-end; that is, it just uses the existing code coverage framework. The only thing going on here is that the tool provides a way to store, load, and display this information. This means that the tool displays coverage for un-compiled files only. We thought about fiddling with this, but finally decided that the existing behavior was probably about as useful as anything else we'd come up with, and a lot more robust.
URL for docs:
http://planet.racket-lang.org/package-source/jowalsh/code-coverage.plt/1/3/planet-docs/code-coverage/index.html
As you might expect, it's a one-line install:

#lang racket
(require (planet jowalsh/code-coverage))

Please let us know about bugs you discover!

Racket on FLOSS Weekly

Our own Matthew Flatt was interviewed about Racket on FLOSS Weekly. Also available on YouTube.

Racket v5.1.1

Racket version 5.1.1 is now available from

  http://racket-lang.org/

• The new racket/stream library provides stream-first, stream-rest, a lazy stream-cons, and so on. Streams are a subtype of sequences, so they work in for forms. Some sequence generators, such as in-range, now produce streams. A racket/sequence library replaces the old racket/stream library.
• The new racket/syntax library contains facilities useful for writing macros. The new syntax/srcloc and syntax/location libraries provide support for manipulating source locations.
• The racket/gui library now supports multi-column list boxes and scrolling panels.
• The new ffi/file library is useful for writing foreign library bindings that cooperate with Racket's security guard mechanism.
• Generators from the racket/generator library can now have formal arguments that are used when the generator is fired up.
• Single-precision floating-point support is now enabled by default. Single-precision floats print differently from their default double-precision counterparts, new primitives convert between the two precisions, and new reader syntax supports single-precision literals.
• JIT improvements include a small change to aid x86 branch prediction on function-call returns, which can speed up some programs significantly.
• Typed Racket:
  • The numeric tower has been entirely overhauled. TR programs can now use more precise types than before, and check more numeric properties, such as sign or range properties.
  • Fixnum optimizations have been improved and should apply more broadly.
  • The performance of the typechecker has been improved. In particular, dispatch on large union types should typecheck much faster than before.
• The Stepper can now step through Lazy Racket programs.
• The racket/future library includes fsemaphore values, the future primitive no longer freezes futures (so a future can spawn new futures), and future log messages are more informative.
• PLaneT development links are now version-specific.
• The 2htdp/image library now includes overlay/align, underlay/align, overlay/align/offset and underlay/align/offset.
• The network protocol for universes in 2htdp/universe has changed, so that v5.1.1 is incompatible with earlier versions.
• The "DrScheme" application (which simply ran DrRacket in the last few releases) has been removed. The "MrEd" GUI executables for Windows and Mac OS X have also been removed, although the "mred" console executable remains for Unix and Mac OS X to support old scripts.

Writing ‘syntax-case’ Macros

Disclaimer: This is not really a tutorial on macros, it's more of a quick introduction to using Racket's syntax-case-based macros for people who are familiar with symbolic macros and miss their “simplicity”. It's also not comprehensive or thorough or complete, it's just intended to provide a rough quick overview of how to write macros. It was originally posted on comp.lang.scheme in a thread called “Idiot's guide to Scheme macros”, but I avoided that title here, since it's not a general purpose guide. (Also, it's yet another attempt to dispel the irrational “macrophobia” some people have when it gets to hygienic macros, leading them back to using defmacro with all its problems.)

The main idea with Racket's macro system (and with other syntax-case systems) is that macros are syntax to syntax functions, just like the case of defmacro, except that instead of raw s-expressions you're dealing with syntax objects. This becomes very noticeable when identifiers are handled: instead of dealing with plain symbols, you're dealing with these syntax values (called “identifiers” in this case) that are essentially a symbol and some opaque information that represents the lexical scope for its source. In several syntax-case systems this is the only difference from defmacro macros, but in the Racket case this applies to everything — identifiers, numbers, other immediate constants, and even function applications, etc — they are all the same s-expression values that you're used to, except wrapped with additional information. Another thing that is unique to Racket is the extra information: in addition to the opaque lexical context, there is also source information and arbitrary properties (there are also certificates, but that's ignorable for this text).

With this in mind, explaining the rest is not too difficult:

• (syntax-source stx), (syntax-position stx), (syntax-line stx), (syntax-column stx) — retrieve parts of the source location information.
• (syntax-e stx) — takes a syntax value and returns the value it “wraps”. For example, if stx is an identifier you'd get a symbol, and if it's a number you'd get the number. If it's a simple parenthesized form, you'd get a list of syntax values for the subforms. Note that the list can be improper, with the last element being a syntax object that contains a proper list. (But the list will actually be improper if the original syntax was a dotted list.)
• (syntax->datum stx) — takes a syntax value and returns the plain s-expression that it holds. This is done by recursive uses of syntax-e. (It would be a simple definition that does what you'd think it should do.)
• (syntax->list stx) — sometimes you want to pull out the list of syntax values from a given parenthesized syntax, but syntax-e does too little (can still return an improper list) and syntax->datum does too much (gives you back raw s-expressions). syntax->list is a utility function that uses syntax-e as many times as needed to get back a proper list of syntax values. If that's not possible (if the input syntax was not a proper list), it returns #f, so it serves as a predicate too.
• (syntax-property stx prop) — returns the given property value from stx, if any, and #f if none. For example, try

  (syntax-property #'[foo] 'paren-shape)

  (The #' is similar to a quote, but for syntax values — I'll get to that later on.)
• Note that there is no accessor for the opaque lexical scope, and as you'll see next, you don't need one.
• To create a piece of syntax you use datum->syntax, and you give it an s-expression which will be the “contents” of the resulting syntax object. (The input can contain syntax values, which are left as is.) But when you do that you need to give it the other bits — including the lexical context thing, which you have no access to. The way that's done is:

  (datum->syntax context-stx input-sexpr)

  This returns a syntax value that wraps the input-sexpr value, using the lexical scope from context-stx. A common way to “break hygiene” and create a binding that is visible to the macro user's code is:

  (datum->syntax stx 'foo)

  where stx is some syntax value that you get from the user input to the macro. It returns a foo identifier that has the same lexical context information as stx, so it's as if it came from there.
  Note that there is actually another optional argument that specifies the source (either using another syntax object, or as an explicit list), and another for copying the properties from — so an alternative to the above would be:

  (datum->syntax stx 'foo stx stx)

  which also makes the source information and the properties be the same as those of stx (for example, this can matter in case of syntax errors ).
• There is also (quote-syntax blah) which creates a quoted syntax, with its lexical source from the place it appears.
• Finally, define-syntax does the magic of tying a name with a transformer function.

And that's almost everything that you need in order to write hygienic (and non-hygienic) macros. Very inconveniently.

For example, here's a simple while macro (use this in a file that starts with “#lang racket”):

(define-syntax (while stx)
  (define subs (syntax->list stx))
  (datum->syntax
   stx
   `(let loop ()
      (when ,(cadr subs)
        ,@(cddr subs)
        (loop)))
   stx))

which breaks like this:

(define x 2)
(let ([let 5])
  (while (< x 10)
    (printf "x = ~s\n" x)
    (set! x (add1 x))))

The problem is that all of those quoted names are getting the context of the user input, which is not the right thing (it's close to a defmacro). To fix this, you need to quote-syntax all of these identifiers, so they'll have the macro source instead of the input source:

(define-syntax (while stx)
  (define subs (syntax->list stx))
  (datum->syntax
   stx
   `(,(quote-syntax let) ,(quote-syntax loop) ()
     (,(quote-syntax when) ,(cadr subs)
      ,@(cddr subs)
      (,(quote-syntax loop))))
   stx))

But that's clearly insane... More than being tedious, it's still incorrect since all of those function application parens will have the user's lexical context (Racket has a special implicit #%app macro that gets used in all function applications, and in this case the context of this application will make it unhygienic). Instead of doing this, a better approach would be to create the resulting syntax with the lexical context of the macro source by changing just that argument:

(define-syntax (while stx)
  (define subs (syntax->list stx))
  (datum->syntax
   (quote-syntax here)
   `(let loop ()
      (when ,(cadr subs)
        ,@(cddr subs)
        (loop)))
   stx))

And that's simple again, and works fine now.

The problem is that it's tedious wrt to deconstructing the input (which happens to be trivial in this case), and wrt slapping together an output value — and that's where syntax-case comes in. It addresses the both by using pattern matching, where identifiers in patterns are bound as “syntax patterns”. A new form is added — syntax — which is similar to a quote, except that (a) it actually quotes things similarly to quote-syntax, with the lexical context of the syntax form; and (b) pattern variables are substituted with what they matched. With this, the above macro becomes much easier:

(define-syntax (while stx)
  (syntax-case stx ()
    [(_ test body ...)
     (syntax (let loop ()
               (when test
                 body ...
                 (loop))))]))

The first line specifies that you want to match the stx input syntax, and that you have no “keywords” (in the same sense as in syntax-rules). The second line is the pattern that is matched against this input — with two pattern variables that match the second subexpression and the sequence of expressions from the third and on. (The first subexpression is matched against _ which is a wild-card that matches anything without binding a pattern variable — the head part is often not needed, since it's just the macro name.) The last line is the output, specified using syntax, which means that it's very similar to the previous version where everything is given the lexical context of the macro and the two pattern variables are replaced with the two matches (so body gets spliced into the resulting syntax).

Now, say that you want an unhygienic user-visible piece of syntax. For example, bind the always entertaining it thing to the test result. This:

(define-syntax (while stx)
  (syntax-case stx ()
    [(_ test body ...)
     (syntax (let loop ()
               (let ([it test])
                 (when it
                   body ...
                   (loop)))))]))

won't work because it has the macro source — it's hygienic and therefore not visible. Instead, you need to use datum->syntax with the user syntax:

(define-syntax (while stx)
  (syntax-case stx ()
    [(_ test body ...)
     (let ([it (datum->syntax stx 'it)])
       (syntax (let loop ()
                 (let ([it test])
                   (when it
                     body ...
                     (loop))))))]))

But this doesn't really work since it needs to be bound as a pattern variable rather than a plain binding. syntax-case can be used here again: (syntax-case <name> () [foo <body>]) will match foo against the <name> syntax, and if it's a name then it will be bound as a pattern variable in the <body>.

(define-syntax (while stx)
  (syntax-case stx ()
    [(_ test body ...)
     (syntax-case (datum->syntax stx 'it) ()
       [it (syntax (let loop ()
                     (let ([it test])
                       (when it
                         body ...
                         (loop)))))])]))

Note that since it is a pattern variable, it doesn't need to be unquoted — syntax will do that.

Finally, there are some more conveniences. First, with-syntax is a macro that binds pattern variables (by a similar translation to syntax-case):

(define-syntax (while stx)
  (syntax-case stx ()
    [(_ test body ...)
     (with-syntax ([it (datum->syntax stx 'it)])
       (syntax (let loop ()
                 (let ([it test])
                   (when it
                     body ...
                     (loop))))))]))

and there's the #' reader macro for syntax:

(define-syntax (while stx)
  (syntax-case stx ()
    [(_ test body ...)
     (with-syntax ([it (datum->syntax stx 'it)])
       #'(let loop ()
           (let ([it test])
             (when it
               body ...
               (loop)))))]))

and there are also #` and #, and #,@ which are implemented by translating them to uses of with-syntax.

Note that the last example uses the lexical context of the whole form for the new identifier, but that's not only the option. You could use any other part of the macro input — for example, you could use the macro keyword:

(define-syntax (while stx)
  (syntax-case stx ()
    [(hd test body ...) ; need the head now
     (with-syntax ([it (datum->syntax #'hd 'it)])
       ... same ...)]))

or the test expression (use #'test). Each of these choices has subtle differences that are especially important when you're composing macros (for example, using a second macro that expands to a while, where the test expression comes from that macro rather than the user code). Demonstrating these things is a popular way to pass the time in some circles, but I'll avoid it here. In fact, a great way to avoid this whole thing altogether is not create unhygienic bindings in the first place. It sounds like doing so excludes cases where you really want to have a new binding visible in user code, but racket provides “syntax parameters” that can be used more conveniently (and less confusingly) — see an earlier post for a description of that. As a side note, these options are a good hint that a hygienic macro system is more expressive than a symbolic defmacro system, where no such choices exist. Creating such macros using defmacro can appear easier simply because of this lack of choice — in the same way that CPP-style string-based macros are “simpler” than defmacro since they're less expressive (just appending lexical tokens, no structural information).

There are other important aspects of the Racket macro system that are not covered here. The most obvious of them is worth mentioning here: Racket separates the “runtime phase” from the “syntax phase”. For example, if you want to try these examples with “#lang racket/base”, you'll need to add (require (for-syntax racket/base)) since the racket/base language doesn't have a full language in its syntax phase.

Roughly speaking, this makes sure that source code is deterministically compilable by having each level live in its own world, limiting macros to deal only with the input syntax only and not runtime values. (For example, a CLOS implementation in this system cannot check the value of an identifier bound to a class to determine how some macro should expand.) This results in reliable compilations that do not depend on how things were loaded, or whatever happened on the REPL.

The important bottom line here is that you get to write macros with the full language available — and phase separation means that Racket is explicitly designed to make running code at the macro level and using it by the compiler as robust as possible, so you don't have to worry about using any complex system as part of your macro. You just need to keep in mind that the macro world is completely separate from the runtime, and the direct benefit of not worrying about weird interactions with compilation and file loading orders.

Languages as Libraries, PLDI 2011

We've just finished up the final version of our PLDI 2011 paper on language extension in Racket. The paper describes how the module system and the syntax system work together to support new languages with new static semantics, such as Typed Racket. Here's the abstract:

Programming language design benefits from constructs for extending the syntax and semantics of a host language. While C's string-based macros empower programmers to introduce notational shorthands, the parser-level macros of Lisp encourage experimentation with domain-specific languages. The Scheme programming language improves on Lisp with macros that respect lexical scope.

The design of Racket—a descendant of Scheme—goes even further with the introduction of a full-fledged interface to the static semantics of the language. A Racket extension programmer can thus add constructs that are indistinguishable from “native” notation, large and complex embedded domain-specific languages, and even optimizing transformations for the compiler backend. This power to experiment with language design has been used to create a series of sub-languages for programming with first-class classes and modules, numerous languages for implementing the Racket system, and the creation of a complete and fully integrated typed sister language to Racket's untyped base language.

This paper explains Racket's language extension API via an implementation of a small typed sister language. The new language provides a rich type system that accommodates the idioms of untyped Racket. Furthermore, modules in this typed language can safely exchange values with untyped modules. Last but not least, the implementation includes a type-based optimizer that achieves promising speedups. Although these extensions are complex, their Racket implementation is just a library, like any other library, requiring no changes to the Racket implementation.

To learn how to implement your own new language in Racket, start with this documentation.

Racket v5.1

Racket version 5.1 is now available from

  http://racket-lang.org/

The most significant change in version 5.1 is a rewrite of the GUI library. Unix/X users will see the biggest difference with this change, because DrRacket and all Racket GUI programs now take on the desktop theme for menus, buttons, and other GUI widgets.

In the long run, Racket GUI programs on all platforms will improve as a result of the library rewrite. In the short run, beware that this first release of a new library will inevitably include a new set of bugs.

Version 5.1 changes in more detail:

• The racket/draw library—which implements the drawing half the GUI toolkit—can be used independent of the racket/gui/base library and without a graphics display (e.g., without an X11 connection).
  The new library has one small incompatibility with the old GUI toolbox: 'xor drawing is no longer supported. The new library has many additional features: rotation and general affine transformations, PDF and SVG drawing contexts, gradients, and alpha-channel bitmaps.
• The GRacket executable is no longer strictly necessary for running GUI programs, because the racket/gui/base library can be used from Racket. To the degree that a platform distinguishes GUI and console applications, however, the GRacket executable still offers some additional GUI-specific functionality (e.g., single-instance support).
  The new racket/gui/base library includes small incompatibilities with the old GUI toolbox: the send-event, current-ps-afm-file-paths, and current-ps-cmap-file-paths functions have been removed. The racket/gui/base library re-exports racket/draw, so it includes the same drawing functionality as before (except for 'xor drawing).
• The new racket/snip library can be used independently of racket/gui/base to work with graphical editor content (e.g., images in student programs). Like racket/draw, the racket/snip library is re-exported by racket/gui/base.
• The Web Server includes a backwards incompatible change that prevents X-expressions and lists of bytes from being directly returned from servlets. This change will increase performance for those types of responses and allow easier experimentation with response types. Please see "collects/web-server/compat/0/README" in the installation to learn about porting your servlets forward. Don't worry. It's easy.
• The new raco demodularize tool collapses a module's dependencies into a single module comprising the whole program. This transformation currently provides no performance improvement, but is the basis for cross-module optimization and dead-code elimination tools to come. The transformation is currently useful for static analysis of whole Racket programs.
• The picturing-programs teachpack, formerly installed via PLaneT, is now bundled with the standard distribution. Use the teachpack with (require picturing-programs) instead of (require installed-teachpacks/picturing-programs). The old PLaneT-based installation procedure still works, but it now merely installs a stub that invokes the bundled version.
• Slideshow picts, racket/draw bitmaps, and images created with 2htdp/image can now be used directly in Scribble documents. More generally, the new file/convertible protocol enables any value that is convertible to a PNG and/or PDF stream to be used as an image in a Scribble document.
• The Same game sports a new look and an improved scoring system. (The current known high score is 12,429; can you beat that?)