Fixed Racket v5.2.1 Installers

Dear Racketeers,

We have just released a DrRacket version 5.2.1 that starts fine today.  The fixed version has replaced the 5.2.1 installers.  This version and the original 5.2.1 differ only in this one fix.

Racket v5.2.1

Racket version 5.2.1 is now available from

http://racket-lang.org/

Release Highlights:

• Performance improvements include the use of epoll()/kqueue() instead of select() for the Racket thread scheduler, cross-module inlining of small functions, and the use of SSE instead of x87 for JIT-compiled floating-point operations on platforms where SSE is always available (including x86_64 platforms). A related change is the interning of literal numbers, strings, byte strings, characters, and regexps that appear in code and syntax objects.
• DrRacket uses a set of composable ray-traced icons available from the new images library collection.
• Typed Racket's typecheck-fail form allows macro creators to customize the error messages that Typed Racket produces. This is especially useful when creating pattern matching macros.
• The performance of Redex's matcher has been substantially improved; depending on the model you should see improvements between 2x and 50x in the time it takes to reduce terms.
• Plots look nicer and are more correct at very small and very large scales. New features include customizable dual axis ticks and transforms (e.g., log axes, date and currency ticks, axis interval collapse and stretch), stacked histograms, and 3D vector fields. The legacy fit function and libfit have been removed.
• The 2htdp/universe library's big-bang form supports an experimental game pad key handler.
• The db library now supports nested transactions and PostgreSQL arrays. Bugs involving MySQL authentication and memory corruption in the SQLite bindings have been fixed.
• The Macro Stepper tool in DrRacket no longer executes a program after expanding it.
• In the DMdA teaching languages, infinite recursive signatures ("streams", for example) with no intervening mixed are now supported, and the signatures of record definitions without fields now have generators for use with property.
• MysterX's ActiveX support is deprecated and will be removed in the next release. MysterX's core COM functionality will become deprecated in the next release, but COM functionality will be supported for the foreseeable future as a compatibility layer over a forthcoming ffi/com library.

Zack Galler's Experience with Stateful vs Stateless Web Apps

Communication using HTTP between client and server is a simple problem of halted computation.

A client computes a request, transmits and halts, waiting for a server response. On receipt, the server computes a response, transmits and halts, waiting for the next client request.

This much is well known.

Racket's magnificent stateful Web server does three things on the server side:

1. it reifies a Racket continuation, capturing where the server computation has halted.
2. it externalizes the continuation, creating a URL-representation that uniquely maps to the Racket continuation
3. it disseminates the externalized continuation to interested clients, typically via HTTP response, but alternately via SMTP or any other protocol.

Then, it waits.

Later, when presented with an externalized continuation, a quick inverse mapping occurs, the underlying Racket continuation is invoked, and the server processes the new client request.

Rinse and repeat.

The problem with this approach is twofold

1. the reified Racket continuations live in server memory. And there's no safe way to garbage collect, as the continuations could be invoked at any time. There are strategies to reclaim memory, but some load level will noticeably decrease the performance of your application. And its not possible to figure out what that load level is prior to finishing your application. This is a problem.
2. Again, the reified Racket continuations live in server memory and cannot be moved. So there's no way to scale an application to more than one server. It's a necessarily one machine system. This makes problem #1 worse.

Racket's yet more magnificent stateless Web server does exactly the same three things:

1. to reify, it rewrites the entire call stack into a format known as A-Normal Form (ANF).
2. to externalize, the ANF'd stack is encoded for transmission over HTTP.
3. and then it's sent over to the client (dissemination).

Later, when presented with encoded stack, the stateless server performs an inverse transform to reconstruct the call stack, at which point the server keeps going.

So we've lost the invocation step and substituted a reconstruction.

But in exchange, we've eliminated continuations from server memory, and solved both enumerated problems above. Neat trick.

I provide a few lessons learned for the archives for the next person to attempt porting #lang racket to #lang web-server code.

First, the predicate serializable? from racket/serialize is invaluable. The #lang web-server code will not transform if there are non-serializable constructs in the dynamic extent of the invocation of send/suspend, such as a local binding or argument.

Second, invocations of native continuations reified with call/cc frequently throw errors related to continuation prompts, such as “attempt to cross a continuation barrier” or “no corresponding prompt tag in continuation”. In all cases, I was able to remedy the situation by enclosing the invocation in call-with-continuation-prompt. This may be an error in the system, but it is unclear at this time.

Third, the transformation does not allow parameters or dynamic-wind, because the internal data-structures representing them are not serializable, but continuation-marks can be used to reimplement the piece of the functionality you need.

Finally, thank you to the Racket team. I think the stateless Web language is important technology and must have required an enormous amount of work to implement.

Anecdotally, application speed seems at or better than the stateful code.

To learn more about the stateless Web application infrastructure, consult the manual or post to the mailing list.

(This post was written by Zack Galler with minor edits before posting by Jay McCarthy.)

Racket v5.2

Racket version 5.2 is now available from

http://racket-lang.org/

Release Highlights:

• DrRacket comes with an experimental, on-line check syntax tool, although this new tool is disabled default. See below for more information.
• The new db library offers a high-level, functional interface to popular relational database systems, including PostgreSQL, MySQL, and SQLite, as well as other systems via ODBC.
• A new XREPL collection provides convenient commands for a plain racket REPL. It is particularly convenient for people who prefer console-based work and alternative editors. See also the new chapter on command-line tools and other editors at the end of the Racket Guide.
• The plot collection has been reimplemented in Racket. It now offers PDF output, log axes, histograms, and more. Some code that uses plot will still work, and some will need light porting. The plot/compat module offers expedient backward compatibility.
• DrRacket uses more conventional key bindings: C-t creates a new tab, C-w closes the current one, and C-r runs the definitions. On Mac OS X, the Command key is used. See “Defining Custom Shortcuts” in the DrRacket manual for an example that uses the old key bindings.
• The new raco link command registers a directory as a collection, which allows the collection directory to reside outside the “collects” tree and without changing the PLTCOLLECTS environment variable.
• Typed Racket:
  • Typed Racket provides static performance debugging support to show which code gets optimized and point out code that does not. Use the “Performance Report” button in DrRacket.
  • More intuitive types in printouts in the REPL and in error messages. Use :query-result-type to explore types, or :print-type for a full printout.
  • Typed Racket now supports defining function with optional arguments using the same syntax as Racket.
• Redex now supports specifying (and testing and automatically typesetting) judgment forms including type systems and SOS-style operational semantics.
• Fixed several GUI problems, including problems on Ubuntu 11.10 (GTK+ 3) and 64-bit Mac OS X.
• Internal-definition expansion has changed to use let* semantics for sequences that contain no back references. This change removes a performance penalty for using internal definitions instead of let in common cases, and it only changes the meaning of programs that capture continuations in internal definitions. Internal definitions are now considered preferable in style to let.
• Support for begin-for-syntax has been generalized; modules may now define and export both value bindings and syntax bindings (macros) at phase 1 and higher.

  Due to a bug, phase 1 syntax (or higher) is not available in DrRacket's #lang-based REPL. A simple workaround is to disable debugging in DrRacket (see “no debugging” radio button in detailed language dialog).

Additional Items:

• The racket/gui library (and Slideshow) provides more support for multiple-screen displays.
• DrRacket remembers whether an opened file used LF or CRLF line endings, and will continue using the same. When creating a new file, a preference determines how it is saved.
• net/url can now follow HTTP redirections.
• The LNCS and JFP class files are no longer distributed with Racket. Instead, they are downloaded on demand.
• The Algol language implementation is now available as a plain language using #lang algol60.
• The Racket-to-C compiler (as accessed via raco ctool or mzc) has been removed; Racket's JIT has long provided better performance, and the FFI provides better access to C libraries.
• Contracts can be applied to exports with the new contract-out form within provide, instead of a separate provide/contract form. (The new contract-out form is implemented as a new kind of “provide pre-transformer”.)
• The date* structure type is an extension of date with nanosecond and time-zone-name fields.
• New looping constructs: for/sum and for/product.
• Direct calls to keyword-accepting functions are now optimized to eliminate the overhead of keywords. In addition, the compiler detects and logs warnings for keyword-argument mismatches.
• The libfit interface is available from plot/deprecated/fit, and will be removed in the near future.
• The Unix installer has been re-done, and it is now more robust.
• The built-in reader and printer support for Honu is removed. (This functionality is re-implemented in Racket.)

On-line Check Syntax:

DrRacket now provides an on-line version of the syntax check tool, which means that syntax checking runs automatically while you continue to edit a program. With this tool enabled, its annotations (e.g., binding arrows) and actions (e.g., the renaming refactoring and direct documentation links) are almost always available.

We have noticed that on-line syntax checking renders DrRacket unstable on occasion, perhaps because it relies on relatively new support for parallelism. Occurrences of the problem are rare, but they are not rare enough, which is why we have disabled the tool by default. At the same time, current users of the tool find it so valuable that we felt it should be included in the release. We expect to track down the remaining problems and enable the tool by default in near-future release.

To enable on-line syntax checking (for #lang-based programs only), click on the red dot in the bottom right of DrRacket's window. To turn it off, click there again.

On eval in dynamic languages generally and in Racket specifically

The eval function is at the heart of a dynamic language, and it strikes many newcomers as an amazingly powerful tool. At the same time, experienced programmers avoid eval, because unnecessary use creates trouble. It’s not easy to explain why eval should be avoided or when it”s appropriate to use eval, but I’ll take another stab at it here.

What is eval?

Consider the following “program” in English prose:

Assume that your favorite color is red. Now imagine a balloon that is your favorite color. Paint a canvas the same color as the balloon.

As English goes, that’s a fairly clear program with a fairly well-defined result. When I follow those instructions, at least, I will always produce a red canvas (assuming that I have a canvas and some red paint, but a potential lack of art supplies is not the point here).

I would come up with a red canvas even if I read the instructions when surrounded by people who speak only Chinese, obviously, since I’m the one reading the instructions. Furthermore, it would be straightforward to translate the program to Chinese, and then a person who reads Chinese would produce a red canvas.

A translator might even take the liberty of simplifying the program to just

Paint a canvas red.

The translation loses some of the poetry of the original, but the result is the same.

In Racket terms, the paragraph corresponds to a module. It can be compiled (i.e., translated) and optimized (i.e., simplified). A program can be made up of multiple modules that are written in different languages, but since each module can be reliably translated, they can all be compiled into some common language to run the program.

Here’s a different program:

Tell the person next to you “Assume that your favorite color is red.” Tell the person “Now, imagine a balloon that is your favorite color.” Tell the person “Paint canvas the same color as the balloon.”

Getting a red canvas back may be a little trickier in this case. If the person next to me speaks only Chinese, then my program may fail with a message-not-understood error.

If I want to translate the program to Chinese, then it’s not clear whether the parts in quotes should be translated. Maybe I mean for a person who can read Chinese but only sound out English to run the program when surrounded by English speakers, or maybe I mean for a Chinese person to run the program when surrounded by Chinese people. Either way, I have to be a lot more specific to a translator. For more complex programs, the instructions to the translator can become complex and fragile.

Finally, a translator probably won’t feel comfortable simplifying the program to

Tell the person next to you “Paint a canvas red.”

because there could be all sorts of environmental conditions that make the result different—such as people who are willing to paint but unwilling to accept assumptions about their favorite colors.

The paragraph with “tell the person...” is a program that uses eval. It can’t be compiled and optimized as well as the earlier paragraph, and the language context in which it is run may change the result. The quotes around sentences correspond to the quote in front of an expression passed to eval in Racket; there’s no particular reason that the language for eval will match the language of the program that has the quoted text. The issues become even more complex if you try to implement different parts of the program in different languages.

If the analogy to multiple spoken languages seems strange—maybe your language is Javascript, period—the problem of translation to another language is really a proxy for program understanding. There’s a direct connection to performance and optimization (i.e., translation to efficient machine code), but using eval also makes a program more difficult to understand for the same reasons that it makes the program more difficult to translate. For example, a reader of your program may not be able to tell whether “assume your favorite color is red” is just a rhetorical device to get to a red canvas or whether some new instructions will arrive that will ask for your favorite color.

When is eval Good?

The program with “tell the person next to you” above uses eval in a bad way. The task could just as well be performed by the person reading the instructions, instead of getting another nearby person involved.

Some other uses eval are both good and necessary. For example, consider the following program:

Ask the construction manager for instructions. Walk to the building site and convey those instructions to the construction crew.

This program uses eval when it conveys instructions to the construction crew, but no quoted forms appear in the program. The absence of quoted code is one sign that eval may be appropriate. Note that the program could work no matter what language the manager and crew speak, although there is an implicit (and sometimes non-trivial) assumption that the manager and crew speak the same language.

Here’s another example:

Go outside, and tell each member of the construction crew “take a lunch break, now.”

There’s a quoted program in this case, but it’s crucial to ask other people to run the quoted program, instead of just taking the lunch break yourself. That is, eval is really necessary. The implementor of this program takes on the burden of making sure that the instructions are in a suitable language, however, and may need to parameterize the quoted program by an explicit action to translate it to a language understood by the construction crew.

Here’s one more reasonable example:

Ask the construction manager for instructions. Follow them.

In this case, it’s the construction manager’s problem to give you instructions in a language that you understand.

Here’s a questionable example:

Decide how long to work before lunch, say N hours, and write a note to yourself to work N hours. Add to the note by telling yourself to take a lunch break afterward.

If you could really write that program without quotes, then it’s probably ok. The example is misleading, though, because languages don’t usually support

write a note to yourself to work N hours

You’d have to write instead

write a note to yourself that says “work” followed by the number N and then “hours”

and the quote marks are where the problem comes in. If you translate the program to Chinese, then you have to be careful to somehow translate “work” and “hours” to Chinese, too.

The point here is not that programs without quoted text are clearly good or that programs with quoted text are clearly bad. The real point is that a programmer has to be especially careful about passing around instructions and using quoted instructions. Using eval means accepting the burden of using instructions will make sense by the time they are delivered. That burdened is best avoided, which is why experienced programmers avoid eval, but some of the examples illustrate cases where the burden is not avoidable or where the actions enabled by eval make the burden worthwhile.

Using eval in Racket

In the context of Racket, the multiple-language analogy is relatively accurate, because Racket is about having many programming languages work together and allowing programmers to define ever better languages and language constructs. In Racket, it’s especially likely that a library written in one language is used in a context where another language is the default.

Newcomers to Racket sometimes stumble over the fact that

 #lang racket
 (define my-x 1)
 (eval '(+ my-x 2))

or even

 #lang racket
 (eval '(+ 1 2))

does not work at all, and yet if the program

 #lang racket
 (define my-x 1)

is loaded into a read-eval-print loop—for example, by clicking the “Run” button in DrRacket and then typing into the lower interactions panel—then

 (eval '(+ my-x 2))

works as expected.

DrRacket’s interactions window has to use eval in the sense that it reads an expression to evaluate and then passes it on to the interpreter for an answer. More generally, to make various pieces of the environment fit together, DrRacket sets eval globally to use the module’s language while evaluating expressions in the interactions window. In Racket terminology, DrRacket sets the current-namespace parameter to the module’s namespace when it initializes the interactions window. In contrast, while the module body is being evaluated, eval treats expressions as being in the language that is empty by default, which is why eval during the module evaluation produces a different result from eval during the interactions windows.

You may wonder why DrRacket doesn’t initialize the namespace of eval to be the module’s namespace from the start, so that in-module uses of eval and the interactions window behave the same. In a program that is implemented by multiple modules, which module’s language should be used? In particular, if the language it’s always the main module’s language, then a module may behave differently on its own than as part of a larger program. In the process of developing Racket and DrRacket, we’ve seen many such problems, and so Racket now arranges for the default language to be empty (which is different from any useful language) to help programmers remember that there’s a language issue to consider whenever eval is used.

The Racket Guide’s chapter 15 covers in more depth the issues and namespace tools of Racket for harnessing the power of eval:

http://docs.racket-lang.org/guide/reflection.html

Think of eval as a power tool. For some tasks, there’s no real substitute, and so we want eval around. At the same time, eval should be used with care. In dynamic languages generally, that means a reluctant and targeted use eval. In Racket specifically, it means knowing the namespace toolbox and being as explicit as possible about the intended context for dynamic evaluation.

Racket v5.1.3

Racket version 5.1.3 is now available from

  http://racket-lang.org/

This is a bugfix release, resolving the DrRacket issue with the contour view. In addition, two tex files with problematic licensing were removed.

Racket v5.1.2

Racket version 5.1.2 is now available from

  http://racket-lang.org/

• The download page includes 64-bit installers for Mac OS X, Windows, and two Debian flavors. Racket now supports OS X Lion.
• Racket now includes a new racket/place library to support parallelism, complementing racket/future. Racket's parallel build process is now based on places instead of multiple OS processes.
  Places support share-nothing parallelism and message-passing communication. Compared to futures, places are heavyweight, but they have a simpler performance model.
• The syntax-certificate system has been replaced by a syntax-taint system. Both certificates and taints were designed to protect otherwise inaccessible bindings from abuse when they appear in macro expansions. Taints are simpler and lighter, and the switch closes known holes in the certificate system. Macros that are not implemented with syntax-rules or define-syntax-rule, however, must explicitly use syntax-protect to protect their expansions from abuse.
• The net/url library supports HTTPS connections, but beware that by default all sites are accepted (equivalent to ignoring a browser's warnings about untrusted certificates).
• Error messages in the student languages use a simplified vocabulary and consistent phrasings. If you maintain curriculum material or teachpacks then please consider updating. See the “Error Message Composition Guidelines” section in the documentation for details.
• Typed Racket: almost all core Racket data structures and operations are now accessible in Typed Racket (most of this work is due to prolific contributor Eric Dobson). The performance of the typechecker has been significantly improved.
• The scriblib/bibtex library supports BibTeX-formatted citation databases in Scribble documents. BibTeX can be tricky to parse, so please report failed entries as bug reports.
• The for forms now support an #:unless clause, and a nonnegative integer can be used as a sequence. The new compose1 function creates single-valued composition functions. The racket/function library now provides identity, thunk, and thunk*.
• The license has been clarified: we now use LGPLv2.1 uniformly. (The license file used to specify LGPLv2, contrary to the download pages.)