Ever notice that at the beginning of a render, you have a stub image written in place of the rendered output image? That stub has readable text that can be used to connect to streaming tile data. This offers you some alternatives to imf_disp, which allows you to watch the standalone rendered tiles come into an image window.

In a recent email on the mentalray mailing list forum, Thomas Driemeyer suggested we repost this here. In addition, we address the follow up question on how to access floating point framebuffer data below.

Basic Stub Data

Excerpted from the "Realtime Image Display (DISP)" section 5.3.2 of Programming mental ray ...

"[Stub data is written] to a socket whose number is written to files. If there are file output statements in the camera, the specified image files are written as 128-byte files containing an ascii line such as:

   ray3.3,720,576,voxel,2478,1.000000,140528,24793.3

"The comma-separated fields contain:

• the mental ray version (2.0 for both mental ray 2.0 and mental ray 2.1, and 3.3 for both mental ray 3.3 and 3.4 since the display format is compatible),
• the width, and
• the height of the image,
• the name of the master host (here, ``voxel''),
• a socket ID,
• the gamma value, and
• the process ID of the master mental ray,
• a second socket ID for framebuffer access^3.3

"The 128-byte file contains -1 as port numbers in mental ray 3.3.3 or higher if firewall mode was enabled by the package that mental ray is integrated in. This mode attempts to circumvent defective Windows ``firewalls'' that consider any attempt to create a socket an Internet access, and block it. mental ray tries to limp along in such a case by avoiding all socket creates. This is a Windows-specific problem, Linux and Unix are fine.

"The protocol used to send data to a file descriptor or to a socket are identical. Data is sent in packets. Each packet consists of a header, followed by optional image data. The header consists of five integers in network byte order (big-endian, high byte first): one packet type code followed by four parameters. The following packets are defined:

Type code	1	2	3	4	data size	function
6	frame	0	0	0	0	start of a new frame
5	width	height	gamma	0	0	image size (file descriptor only)
2	xl	xh	yl	yh	4*(yh-yl+1)   *(xh-xl+1)	new RGBA image tile
4	0	0	0	0	0	end of image
3	0	0	0	0	0	refuse connection

"Image size packets (type 5) are sent only to file descriptors and not to sockets because socket readers have obtained that information from the 128-byte file already. The gamma value is a floating-point value, not an integer. mental ray 2.1 and 3.0, but not 3.1 and later, refuse connections (type 3) if there is already another connection; this can happen only during socket connection attempts. Packets with types not listed in the table above should be quietly ignored, they will consist of only the header.

"Only image data packets (type 2) have data after the header. Data is sent in RGBARGBA... order, with eight bits per pixel component, beginning with the lower left corner of each tile. The coordinates of the lower left tile pixel (xl, yl) and of the upper right tile pixel (xh, yh) are in absolute image coordinates. If output shaders run, the image may be sent multiple times, once as rectangular tiles during rendering and then once per output shader in blocks of scanlines. The order of tile and scanline packets is undefined.

Thomas notes ..

"... with mental ray 3.3 you get a second port listed at the end of the 128-byte stub file line. If you open that, you can list available frame buffers, and request raw frame buffer data. Conversion is up to you. I am afraid this isn't documented; try starting a long-running render and connect to the port with telnet (or the browser of your choice, using telnet://renderhost:port), and ask for help. It's very brief but complete.

We'll walk through an example of this in the next section. But before that, let's make a documetnation attempt, derived from protocol information I got from Thomas...

The socket connection protocol for the second port allows access to the data streaming out from any defined framebuffer. (Note that this is framebuffer image pixel data, after the samples have been filtered/collected into pixels.) These commands are sent to the thread handling image display, which replies to the socket using the packet protocol defined above. Here are the commands available for sending to the second port:

version	returns protocol version, ray version, ray date, host type
fb_list	returns for each framebuffer:    ID, type, xres yres, components bits/comp, name    ... IDs are miRC_IMAGE_* from shader.h, types are miIMG_TYPE_* from shader.h. Note lack of comma for resolution and pixel spec.
stream_begin IDs	stream the identified framebuffers, IDs are a comma separated list
stream_end	stop the stream
help	lists out these commands

If you send a stream_begin command, the stream replies with frame and rectangle packets as previously described, in other words:

frame_begin:	framenumber xresolution yresolution
rect_begin:	tile_xl tile_yl tile_xh tile_yh
rect_data:	binary_data_size, tile_xl tile_yl tile_xh tile_yh, framebuffer_id framebuffer_type, xresolution, yresolution, components_per_pixel bits_per_component
rect_end:	tile_xl tile_yl tile_xh tile_yh

The following interaction demonstrates the protocol when connecting to the second port. Green indicates socket reply. The red number is our port. Note how the help and fb_list replies end with an empty line after the colon.

Framebuffer Access Example (aka, Fun With Sockets)

OK. Let's learn by doing. For this example, we use a standalone ray on a Linux machine and an mi file that renders long enough for us to poke around on telnet (the SSS mi file of grapes submitted by Dominika Waclawski soon to be seen on the SSS tips page). As above, green indicates socket reply.

Now let's render to a file called foo.jpg:

[bart@hana2 sss]$ ray -o foo.jpg grapes.mi

Now, a bit after it starts up rendering there will be a stub file at foo.jpg. From another shell, look at the stub data with more:

[bart@hana2 sss]$ more foo.jpg   
ray3.3,640,480,hana2,35126,1.000000,2188,35127

First we see the version, then the resolution, the render host (hana2), the primary port number, a gamma value, the render process ID (pid), and the secondary port number for framebuffer access. Note that if you're rerendering over the top of an already-rendered image, and you run more too early, you'll see the binary image data from the previously rendered image.

If we were to telnet to the first port, we'd just see a bunch of streaming data. The secondary port gives us a little more interactivity. So, let's telnet to it (that last number):

[bart@hana2 sss]$ telnet hana2 35127
Trying 127.0.0.1...
Connected to hana2.
Escape character is '^]'.
help
help: version : return protocol version, ray version, ray date, host type
help: fb_list : return frame buffers: ID, type, xres,yres, components, bits/comp, name
help: stream_begin : set comma-separated list of frame buffer IDs to stream
help: stream_end : stop stream
help: help : this help text
help:
version
version: 1, 3.4.0.566, 28 Sep 2004, Linux 2.4.7-10 (RedHat 7.2, glibc-2.2.4-13), i686, gcc3
fb_list
fb_list: 0, 0 +rgba, 640 480, 4 8
fb_list: 12, 2 +rgb, 640 480, 3 8
fb_list:
^]
telnet> quit
Connection closed.

While conntected, you can try the 'stream_begin 0,12' yourself, as you can then see the data streaming out to you. While its streaming out, you can type stream_end to stop it.

Here's some data streaming fun. Try opening an imf_disp display of foo.jpg, before opening the telnet connection. Then, watch the data stream out to both at once. Keep in mind that the data streaming out to the imf_disp display is the data coming out of that first port. If you telnet to that, you'll be seeing a stream of data mirroring what is going into the imf_disp window.