Sunday, May 24, 2009

Compile This !

I've been looking into compiling high level RenderMan Shading Language (SL).

The shader virtual machine is already more or less there.. with some notable features missing (conditionals for one) ..but I think I need to hurry up and start supporting the high level language. If anything, to have an estimate of the complexity of the work.

RenderMan's SL looks like this:

 float Ka = 1;
 float Ks = 1;
 float roughness =.1;)
 normal Nf = faceforward(N, I);

 Oi = Os;
 Ci = Os * Cs * (Ka*ambient()+Ks*specular(Nf, -I, roughness));
..and my assembly, currently, looks like this:


 Cs  parameter varying color
 Os  parameter varying color
 Ci  parameter varying color
 Oi  parameter varying color
 N  parameter varying normal
 I  parameter varying vector

 Ka  parameter uniform float 1
 Kd  parameter uniform float 1

 Nf  temporary varying normal

 normalize $v0  N
 faceforward Nf $v0 I

 diffuse  $v0 Nf  // -- Kd * diffuse( Nf )
 mulvs  $v0 $v0 Kd

 ambient  $v1   // -- Ka * ambient()
 mulvs  $v1 $v1 Ka

 addvv  $v0 $v0 $v1 // +

 mulvv  $v0 $v0 Cs // * Cs
 mulvv  Ci $v0 Os // * Os -> Ci

 movvv  Oi Os  // Oi = Os

When it comes to implementing, there is a world of differences between the C-like high level shaders and the asm one.

Following the suggestions of the Production Rendering book, I started looking into Lex & Yacc and Flex & Bison, also ANTRL and finally Flex++ and Bison++.
I looked into tutorials, tried to make those tools work, but eventually gave up.

There are two things that really frustrated me. First the availability and the usability of the tools.
ANTRL and Flex++ and Bison++ are supposedly C++ friendly, but I couldn't quite understand how. I found some examples, one seemed good, with some comments in French (?).
ANTRL has this ANTRLWorks Java application that lets you experiement with some nice graph rendering. You can specify options to handle C++ code, only some odd error message comes out.

I found a tutorial on ANTRL which seemed nice, also explaining how to integrate it with Visual Studio, but then it was for version 2.7.8 (or something like that) while the dir structure of the latest version was completely different.

Lex/Flex and Yacc/Bison are tried and true Unix tools, but the Windows versions aren't distributed too officially.. meaning that one has to find a site of some guy that built the binaries for you 8)

The other thing that frustrated me is that those tools are not easy to use at all. I spent some time trying to get my head around one of the many tutorials out there. They all say things along the lines of "Thank God you don't have to parse things yourself !" ..and then they start going about rules, lexemes, with some sort of arcane notations. And I feel like I'll never quite understand what is going on if I don't actually try to code the thing myself.

So, I put the parsing tools on the side and I started writing my own shading language parser.
I'm still only barely extracting tokens, but at least I feel like I'm making a progress. I mean, if I have to spend days or months of my time learning some ancient Unix tools, I'd rather invest that time to understand the basics and code it myself.
How hard can it be to write a very basic C-like compiler ? Not easy for me, but it's been done a thousand times over and lexical parsing, while tedious, is really the easy part.

I think that those compiler writing tools are essential if one needs to periodically write new parsers and syntax analyzers. In the long run, the tool generalizes much of the low level work.. but I'm not in the business of parsing different types of languages or creating new ones, and I can use some experience of compiler writing.

I think I can do away without those tools for now, but I can certainly use all knowledge there is on the subject. The must-have book on compiler writing is "Compilers: Principles, Techniques, and Tools"
  aka the "Dragon Book".
I remember the book (English edition !) in store in Shinjuku. At the time I was interested, but still not quite sure. Also the book was sealed, I could have asked to look into it.. but I really didn't need it at the time. I did need it today however and so I downloaded it (^^;)
Well, if I find it useful I'll probably buy it. If anything to have it in a portable and easy to browse format (portable as into the restroom, at 1038 pages, I don't see myself bringing it on the subway 8) .. but for the time being it was good to find it online.. perhaps illegally, but not unfairly, as having the PDF now made me want to own the book.
..ahhh.. some day we'll look at the stupid anti-torrent policies just like we look now at religious inquisitions of the Dark Ages.

Back to the book !