Fresh air

Nathanael Gentry, 04/08/2020 | Source: ScummVM on Nathanael Gentry

Hello!

This will be short, as last week was rather slow. I mostly finished shoring up all the areas where Director palettes can be tested and set – in the frame palette channel, with the puppetPalette command, with the movie-wide default palette, and also with the palette of cast Lingo. As I look back at my commit log, I realize that this took me many more commits than I expected. There are still some issues that some of our new test targets, including the 1991 title The Riddle of the Maze, recently revealed. As of this morning, though, this game’s elegant artwork has been restored:

Apparently, before Director 4, palette castmembers were forced into the same order that they are stored. If you reorganize them in the cast window, they will actually revert to a default position on save – and this position seems to be in the order of palette creation. Also, Director 3 Macintosh format has a slightly different way of storing palette channel information in the frame. (I have still to push the patches for titles that have this variation.)

I also finally got some of the titles that @trembyle had mentioned on the Director wiki, including the aforementioned Maze. @trembyle has done some immense work testing many new titles for us, and today I enjoyed investigatinCEg some new bugs in new titles. Playing with these new games was a nice breath of fresh air from the targets that we had been testing since the beginning – like Spaceship Warlock and Chop Suey and The Apartment.

My other big task last week was working on the text selection interface. When I think back, I believe my first experience with Director happened when I was pretty young; it was a typing game that also depended heavily on text selection. I remember this game as Director because it crashed once and I distinctly recall the Director projector icon. @sev had done most of the algo work in the MacGUI side, but I needed to write a bit more interface and do lots of testing to discover what was behind a selection offset offset bug.

Once I finish up the palettes and add a few new targets that I found, I will be working on supporting movies that have >8-bit colour. If you look in the graphics code, there are tons of const byte * declarations everywhere, so those will be changed out for a more general implementation. There are also bitmap decoding issues with greater colour depths that I need to look into. This task, a fairly big one, will be a nice way to round out my GSoC work on our Director engine.

A Branded Desktop

Dylan Servilla, 03/08/2020 | Source: Dylan Servilla - ScummVM Blog

It’s been a hectic week since I attended two online conferences going on at the same time, and I had to prepare a talk for one of them, but I still have some exciting progress to share!

This week I got desktop/window border textures working, so the environment surrounding the Director movies looks closer to how it would have back in the day. Though the ScummVM branding would’ve been Mac OS branding, of course. Desktop with Mac OS style windows and a ScummVM desktop background

This was a real pain to get to, since I needed to make all these textures myself, and our MacWindowManager’s hard-coded defaults never work quite right for the Director engine’s use cases. But now it (mostly) works. I just need to fix up a few bugs and rewrite the desktop background code to make it more efficient.

On Friday, we met with John Henry Thomspon, the inventor of Lingo, to show off this and our other progress. It’s always nice to hear the thoughts of such an important Director developer. :)

I also rewrote my Lingo decompiler, ProjectorRays, in C++ so that it can more easily share code with ScummVM, and I’m working to bring it in line with the Lingo discoveries we’ve made while working on the Director engine. I just got it working on Director 4 movies! The output for these older movies isn’t great yet, but that will quickly change.

A huge jump of progress, & back at square 1?

Aryan Rawlani, 01/08/2020 | Source: Google Summer of Code with ScummVM

Hello! What a busy week it has been, and I have some good things to share. In just over a little 3 days of time, I worked for a lot of time to implement the new idea I mentioned I had last week, and it was actually working quite well! I managed to implement the entire basic structure of it, as well as port over 8 of ScummVM's 79 engines to adapt to it!

So, what's the deal with this, and what did I do? The "idea" was just me getting a basic understanding of how things "could" work in a way.

Well, we were at the "createInstance" bridge where I left off last week. This is a member function of either a MetaEngine or an AdvanedMetaEngine class. The problem with this was this had engine code in it, so it couldn't be included in the executable. The basic idea around this was simply to remove it as a class - member function, and instead mark it as an export symbol. What we could then do, is in base class, provide a default functionality. These functions were a Pure Virtual Function, so I simply removed it in order to get what I wanted to achieve.

The "createInstance" method though, would only be exported if we were using a Dynamic-Plugins setup. If the entire engine would be to build into the executable, there wouldn't be any point in doing this, so for those, we still override the base class's function.

If a child class overrode this function, the vtable generated would point correctly to the createInstance method, and now the static plugins work as intended. If a child class didn't override this function, that would mean the class would use it's default implementation wherever used. What could we possibly do here? That's right, we can simply hook up plugins to find a symbol in the engine libraries we have loaded.

Before a game is actually started, we unload all engine-type plugins to save memory and then try to run the game. Running the game usually has the first step of instantiating the engine, so normally when we actually reach to the point where the default "createInstance" is called, we only have 1 engine plugin loaded as a library.

Then, the default implementation basically consisted of fetching this plugin and trying to load a symbol. We have 3 main plugin classes in ScummVM, "Plugins", "StaticPlugins" & "DynamicPlugins". When I fetch a plugin and try to call a function, it would need to be present in the relevant plugin interface, obviously. So, I made pure virtual functions in the Plugins class and overrode those in the Static & DynamicPlugins classes. This way, when I have a "Plugin *plugin", and I use a "plugin->tryInstantiateEngineFuncSymbol()", it looks up in the vtable and proceeds to call the one in DynamicPlugins, as it handles everything dynamic-related things.

Over there, the process is very simple. We find the symbol, and if we can find it, we call it. This way, the engines are instantiated, and then everything proceeds as usual because the bridge has been crossed.

Since we now have a difference between MetaEngines & Engines, they are given different PLUGIN_TYPES, which is basically an identifier. To implement all of this though, I had to make some more changes for the existing functionalities.

For example, we have a unloadPluginsExcept method, which is run before a game starts, and as I mentioned earlier, only keeps a matching plugin in memory. This method takes in a PLUGIN_TYPE, and a plugin itself.

Normally, we just fetch everything in memory of the type mentioned, and comparison between pointers would tell us which one to keep. But, since we now differentiate between MetaEngines & Engines, this method gives in a TYPE_ENGINE and a pointer to a MetaEngine plugin. The way to solve this is pretty simple, we have a method in metaengines called getEngineID. We reconstruct the name of the engine plugin with the help of this, add prefixes and suffixes if necessary (for example on windows, it becomes engine.dll) and then simply fetch each plugin in memory of the type ENGINE, try to match it with our generated filename. This successfully lets us achieve what was intended.

With the basic structure complete, I thought now was the time to go engine by engine and incorporate the new changes. The 8 engines I mentioned were ported earlier were - Agi, Plumbers, Drascula, Dreamweb, Lure, Pink, Sky & Scumm.  There were some things that would need shifting around - we just place the statically linked MetaEngine code into one file. After that, do the whole checking thing for createInstance, exporting it or not etc... For the Scumm engine though, there was a bit of an issue which needed a little more working around.

Great! Everything seems fine, so what's with the "back at square 1" in the title? Well, as it turns out, createInstance is not the only bridge that would need to be crossed over. If you remember when I mentioned what the MetaEngine does, is it usually handles all metadata and things normally the engine wouldn't do. The engine takes care of running the game, while the MetaEngine for other helper stuff. This helper stuff includes instantiating an engine, but it also handles listing saves, creating thumbnails, and querying for metadata for the savefiles, and many more.

It includes listSaves related things, which can have some engine dependencies, something I found when I was looking at the Sword25 engine. What it does, is basically creates an entirely new class, which itself has many engine dependencies, so obviously it cannot be linked statically. So, this is another bridge.

It's different for each engine though. In the 5 engines I ported, each had bare minimum for saving related things, so some shifting around of code made it work well. What that meant is even without loading a plugin, or having one entirely, we could still see savefiles, along with the thumbnail if it supported. It's not the same for all engines, as I pointed out earlier with Sky25 it's a whole different thing altogether.

But wait, with the basic structure done and solving the createInstance problem couldn't we do the same thing with these as well? Well, yes. Implement the base variant in the class, which calls DynamicPlugins to load the symbol and call it. We would then need the individual MetaEngines to not be defining it as a class function, so what I did with createInstance could be replicated and would work.

However, the main issue with this is that we have to include it as a class function if using static, otherwise like a normal function, so the code itself looks like

---
#if PLUGIN_ENABLED_DYNAMIC(ENGINE)
extern "C" PLUGIN_EXPORT function signaturehere {
#else
class function signature here {
#endif
...
normal definition of function
...
}
---

This doesn't look nice, does it? If it was once, maybe it would've been ok but to do this as many as 3 times would be a bit too much, and code starts looking pretty weird.

So, recent discussions Eugene has pointed out that I could actually try to separate the MetaEngine class itself, so detection related things will be placed in one MetaEngine, while the dynamic things in another. We have so many engines, that I probably need to scope out many more of them and understand how this will work well. For now, I have the basic structure in mind, which involves adding 2 new classes (because we have a MetaEngine & AdvancedMetaEngine), but the work could be done with the addition of just one more class was suggested as well, so I'll have to keep that in mind.

The existing hierarchy/structure is something like this:
- We have PluginObject.
- MetaEngine derives from a PluginObject.
- AdvancedMetaEngine derives from MetaEngine.
- Plugins have as a member variable, a pointer to a PluginObject.
- A PluginProvider, which basically helps in creating the plugins.

The new additions will be 2 new classes, and they'll be related to MetaEngines.Those 2 classes will be called (or at least for now, because the name must be nice enough for everyone) MetaEngineConnect (inherits PluginObject) & AdvancedMetaEngineConnect (inherits MetaEngineConnect) so something like the original structure is replicated. These will then have functions like "createInstance, queryMetaDataForSave, listSaves" at least. I'm not sure if that's exactly going to be it because I have more thinking to do.

So, perhaps not back at square 1 but just the square ahead of it maybe? The engines already ported would need reworking, and many more things, so I just shifted to a new branch today and cherry-picked some of the commits from the old branch, which I think would be helpful. There are 79 engines in ScummVM & 5 engines in ResidualVM, so I best get back to work now! Phew, much more work to do!

That's the main thing I'm focusing this week, division of MetaEngine classes, and connecting everything once again properly! This possible way could be further improved and because this is something that will touch all engines, I should be more careful than ever in my thought process. That's it from me for now, see you again next week with some new progress on this hopefully!

Thanks for reading!

Finishing basic shader rendering

gunnar, 29/07/2020 | Source: GSoC 2020 ResidualVM

Now we can render our WME games even with shaders. The most work was actually related to setting up the surrounding code and not so much the actual shaders (minus the one which is responsible for rendering .X models). At the moment, things are somewhat unorganized however and the current code should be refactored, interfaces improved and common code factored out. I would claim that at the moment one can see the fact that WME was designed for the old fixed function pipeline and that the addition of the shader renderer is not completely straight forward. In contrast changing from Direct3D to OpenGL was easier, the only real change that I made was using the OpenGL matrix stack, which now turned out to perhaps been the wrong direction. Well, it can be reimplemented (and is so at the moment), but the old WME code didn’t rely on it either, so I might change this back.

I hoped that lighting done by shaders would produce the same results as the original code but this is not the case.

Let me just assure you that Rune is not supposed to look like he is almost getting sick. Apart from that though, things seem to look just fine.

Below the surface

Nathanael Gentry, 28/07/2020 | Source: ScummVM on Nathanael Gentry

This week, I worked on implementing a few Lingo commands that required some significant backend work, but not much in the rendering code where I have spent so much of my time. I mentioned that I had worked on a custom cursor implementation, and I spent the first part of the week fixing bugs there. Now cursor bitmaps like the starfish on the island are properly displayed at the proper position, whereas after my initial work just a black box showed.

Early in the week, I fixed a subtle rendering bug that had plagued Spaceship Warlock for a while. You’ve seen how the Stambulian policemen in Warlock were bright green; well, this was about the same as the keying colour that the MacGUI was using for border transparency. Once I spotted this, the fix was easy – and also prevented a redundant surface copy. (Yay for performance improvements!) Here you can see how the erroneously applied transparency actually gave a nice look to an otherwise bland wall in Stambul:

I also finished the implementation of custom colour palettes. For a long while the opening to Chop Suey looked all psychedelic. It turns out that a recent refactoring swapped around the order of palette loading so ScummVM was given the wrong palette to use. My partial fix for this revealed that our recently-added target Majestic used a castmember palette, which had been loaded before – albeit improperly. So over the weekend I expanded our Director palette manager to support custom palettes alongside the half-dozen default Mac ones. This also brought along basic support for the puppetPalette command, which controls the palette from Lingo. Now, Chop Suey and Majestic are looking quite handsome:

This project came later in the week, however. I first implemented sound fading, which more intense than I first knew. Since multiple movies can be running at once, my knee-jerk blocking fade loop didn’t work out. Instead, I needed to integrate the sound fade with the existing score stepping methods. I’m amused that that this was one of my largest commits for the week. (Thankfully, though, transitions do seem to block in the original – that would be a pain to refactor.)

Finally, I scratched my head for a while on an issue that @sev has since begun looking into. An interesting Director target is Macromedia’s own guided tour, which takes you “behind the scenes” at their studios to introduce new features of Director. Our renderer implicitly assumed that bitmap sprites had the same dimensions as the underlying castmembers, unless the dimensions had been modified from Lingo. Well, our smartly-dressed friend from the guided tour dismissed that theory:

What’s on the left is the original sprite (plus arms), and on right is the original castmember. ScummVM draws them both the same size. I would never have noticed, except that the sprite for his moving mouth appears all out of place when he isn’t the right size. Director does indeed have an option to scale individual sprites in the score, and we were reading this information in for bitmaps… but when I tried to use it there were puzzling discrepancies in the dimensions we expected and what the file clearly said. I still haven’t figured out why.

It’s been a good week over all, with interesting tasks both in and outside GSoC. You’ve perhaps seen on my bio that I like playing music. One of my friends recruited me to play piano for her upcoming violin competition, so when I need a break from coding I spend a few hours with the interesting sonorities of the great American composer Samuel Barber. And, at university on the weekends, I am working on using machine learning to control dielectric elastomers – smart materials that show much promise for soft robotics.

Events and Things

Dylan Servilla, 27/07/2020 | Source: Dylan Servilla - ScummVM Blog

I spent last week fixing more Lingo issues, mostly event related:

  • Previously, event processing didn’t take into account the fact that multiple movies could be running at the same time, as it was implemented before MIAWs. Events were handled by whatever movie was currently being stepped, even if the event should’ve gone to another movie. I refactored the event handling code, and that’s been fixed.
  • I made the Lingo interpreter pause every 100 operations to process events, preventing long Lingo scripts from blocking mouse position updates and quit events.
  • I changed menu item callbacks to use the event system, whereas before they just haphazardly created and called global functions.
  • I modified the Lingo interpreter so that handlers whose return values aren’t used don’t leave garbage on the stack.
  • Lots of small bug fixes, and a few Lingo features like the title of window

Nothing I did last week is particularly interesting to look at, so here’s a bug in Chop Suey I fixed today: Mud Pup wearing assorted clothes

Before, you couldn’t put clothes on the dog. Now, you can put clothes on the dog. Problem solved. :)

Touching upon all 3 tasks

Aryan Rawlani, 27/07/2020 | Source: Google Summer of Code with ScummVM

Hello! So, this week I have done things that ranged across all my tasks. Recently I checked out our sister project, ResidualVM. ResidualVM's codebase is largely similar to ScummVM, but the main difference is the addition of capabilities to run 3D games. They also take regular snapshots from ScummVM to keep up-to-date. So, the work of RTL GUI which was merged in ScummVM was also present there.

When I was running the application with the Hebrew language, some popups and drop-down buttons looked quite different from the normal GUI. Upon a little investigation, it looked like paddings were being set wrong for the RTL-widgets set in the theme. This was because back when I was on my RTL task, I temporarily set a padding of "2" as a placeholder to check, and then completely forgot about it as the difference was ever so subtle. I'm glad though, that I found this, and opened a PR to fix it.

English, proper paddings


Hebrew - Improper paddings



Hebrew, mirrored and matching padding to English


Last week I left off saying Eugene had some comments about my U32 task, so this was what I worked on next. After solving the review issues, I decided to review my PR more carefully once again, because many issues were being raised. 

This time, I picked up on many things that could also be done and saw some things I had left out.

For example, when a user wants to add a game, they can choose a folder from a file browser. This file-browser can also be a native system open dialog, or a ScummVM custom browser.

Native-windows dialog. The title, and the buttons for "Choose/Cancel" are in english.

So, for windows, ScummVM implements a small layer of Win32 API to implement this. The dialog has some labels which are also set to the GUI language.

Before, I had written some sloppy code (1 line of code change) and expected it to work to as it used to, without thinking too much about it. It was clearly very wrong, and the translations in the dialog were broken. Nevertheless, I worked on adding support to it properly. I already had text in UTF-32, but our Windows wrapper code didn't have anything to convert the text to an LPWSTR (16-bit Unicode characters). I looked up documentation online and followed it, implementing a UTF8ToUnicode. We have a conversion function for UTF32 to UTF8, and then by using the above function (internally calls MultiByteToWideChar for UTF-8 encoding), it was converted properly to LPWSTR. The implementation now looks proper.
Russian translations for dialog title and "OK" button (The OK button has the label - "Choose")
 


One of the issues raised was that I was converting a Unicode string back to a normal string because a class called StringTokenizer was involved. I don't know how I missed that, so I implemented a Unicode string tokenizer.

A StringTokenizer simply takes in a string and some delimiting characters and separates the words based on the delimiters present. So, If I have the string "Hello, World - Welcome.... to ScummVM" and provide the delimiters as - " ,-." - I will get the individual words ("Hello", "World"....), which I can simply search, for example, in a list - if it contains those words or not.

One example where they're used is in the game search filter. Let's say I have a game - "Reversion: The Escape, PC" in a list of 100+ games. If I type in either of those words, the search list should show this game. With everything going Unicode, now what would happen is if you have a game title with non-ASCII chars, like かわいい- and you start typing in か, the game with the title かわいい will show up in the search list. I'm not sure if we really support typing in with any other languages in ScummVM right now, but it is now future-proofed I guess!

Those were pretty much the main things I did for the U32 task. I also cleaned up the PR quite a bit, reviewed it with care so small things get out of the way. 

Yesterday and today I was able to shift focus again to the detection task. The very first thing I did was to drop the previous few commits because the approach was wrong.

So, each engine has an overall engine template. Anything related to detection, the "MetaEngine" or any other related function is supposed to be defined here. It also contains function definitions from classes not declared here. So, my first guess was to restore the function definition to the original files. Say, "example.h" had the declarations, so "example.cpp" will have all the definitions. That's what I did, but I failed to recognize that functions were not defined in the so "example.cpp" for a reason. All detection related code from any classes is grouped over at detection.cpp, so the code overall has a nice structure to it. This also makes it easier for engine maintainers.

After a discussion, what I will now try to do is take code from "detection.cpp" and split it into another file - "detection-static.cpp", which the name suggests, will always build into the executable.

Finally, once again I arrive at the "createInstance" bridge. If you don't remember from last week, a small recap: createInstance is basically a function inside a MetaEngine - which instantiates the proper engine. So, if I am trying to launch the "Plumbers" game, it will instantiate the "Plumbers" engine.

The difficulty with this is that the function itself contains code that is not available in the executable. What I mean by that is,

Say I have an engine, which is used to run the game "exampleGame". This engine is named "exampleEngine".

When we request ScummVM configuration to build this engine as a dynamic module (explicitly linked run-time library), all code in "exampleEngine" goes into the "exampleEngine.dll" file.

Now, the previous dynamic system, everything was in the external library, so we could just load it, and then have "createInstance" instantiate the proper engine. i.e in this case, "engine = new exampleEngine". But now, the MetaEngine is always included in the executable, so that's a problem.

Why a problem? If I include "createInstance" class method inside the executable, it also has the line
"engine = new exampleEngine();"

What is exampleEngine? How will the compiler/linker recognize that, if the engine itself is in an external library? That's the issue of why this particular method cannot live in the executable.

So, what I am doing now, is basically making the createInstance method live outside the executable, and inside the .dll, while still making everything work as it used to.

I tried to approach this in various ways, but most of them had a flaw somewhere. Yesterday, I thought of a good approach, which will make minimal changes across engines and should work correctly. It's still a work in progress though, and no point in expanding on something that could be wrong. So, more on this next week!

I'll basically be focusing on implementing a way to bridge this gap for the next week, & That's it for this week!

Thanks for reading!

Starting rendering with shaders

gunnar, 22/07/2020 | Source: GSoC 2020 ResidualVM

Ok, what’s so special about the following picture:

Well, nothing, except that the scene was rendered with OpenGL shaders instead of the old fixed function pipeline. Which in this case (2d graphics) was not the biggest change, essentially only the 2d projection matrix has to be setup for the shader. 3d graphics will require more work, though. Also some refactoring will be necessary, some of which has already happened. For example, there is a 3d renderer interface class now with two implementations, which can be selected from the ResidualVM menu (a third one, based on TinyGL, is supposed to follow at some point in the future).

For some reason, when walking into the house and also when leaving it, during the animation the cursor becomes completely black, but only with the shader renderer. Now I am not too unhappy about this, at least it shows that the two renderers indeed operate differently.

Speaking of different behavior, the shader renderer might also be helpful when going back to lighting, since currently, the fixed function renderer does not give the same results as the original Wintermute Engine. If the shader renderer can reproduce WME lighting, than something would be wrong with the fixed function settings. If, on the other hand, shader and fixed function renderer produce the same results, something will be wrong with the rest of the code, possible normal vectors. If all three implementations are different, than it’s kitchen sink, though.

Learning the Lingo

Dylan Servilla, 21/07/2020 | Source: Dylan Servilla - ScummVM Blog

This week I worked out some longstanding Lingo issues!

Duplicate Scripts

The first issue was duplicate scripts in Director 4 movies. Each cast member should have at most one Lingo script associated with it, but we were running into movies in which a cast member seemingly had several scripts. There was no obvious way to deal with this - redefining the script usually led to incorrect behavior, and so did keeping the original definition.

These duplicate scripts were rare in most movies, so the problem went ignored for a while, but in our recently added target Majestic Part 1: Alien Encounter, there were several hundred scripts, and almost every one conflicted with another.

Initially, I thought that there must be something that indicated certain scripts, or at least certain handlers within these scripts, were unused. The first place I investigated was the script’s “handler vectors.” These differed between some of the duplicate scripts, and I thought they might hold the key to how the script conflicts should be handled.

“Handler vectors” were identified as an array of 16-bit integers in Anthony Kleine’s Director documentation, but there was no explanation of their purpose. I got in touch with Anthony, but he couldn’t remember what they were for, and they remained a mystery to me for weeks. Once I began deeper investigation, it quickly became apparent that the “handler vectors” are just used to map event IDs to handler IDs. Totally unrelated.

The next suspect was the Lingo context, a container which maps script IDs to script data:

Section LctX {
	Struct header {
		...
		Uint16 [big] freePtr
	}
	...
	Array scripts(count) {
		Struct scriptLink {
			Uint32 unknown
			Uint32 [big] ID // use MMAP!
			Uint16 [big] used // 0 : unused , 4: used
			Uint12 [big] link // For unused entries: link to next unused, or -1.
		}
	}
}

(Source: Brian151)

The two areas of interest are:

  1. The script entry’s used field
  2. A linked list of unused scripts, which begins at the script entry at index freePtr. The entry’s link field gives the index of the next unused script, or -1.

However, after much investigation, it seems that there is actually no difference in how a script with used = 0 and a script with used = 4 should be handled. The linked list does indeed indicate unused scripts, but all of the entries in the list seem to have an ID of -1. Thus, these unused scripts have no script data associated with them, and they were never being loaded in the first place. There was no way they could be causing conflicts, since they didn’t really exist.

To finally solve the mystery, I had to throw out what I thought I knew about how Lingo scripts are linked to cast members. For years, the Director reverse engineering community understood that:

The scripts are not owned by their individual Cast Members in the Key Table [which links cast members to most of their assets] as you may expect. Instead, each Lingo Script has the number of its corresponding Cast Member (Source: Anthony Kleine).

After a few days of testing, I noticed that the cast member ID stored within Lingo scripts was sometimes incorrect. Or, as in Majestic, almost always incorrect. There had to be some other way by which cast members were linked to their scripts.

The obvious place to look was in the cast member data, which is split into two parts - data specific to the cast member type, followed by largely standard cast member info. We had previously identified a scriptId field in the data specific to script cast members, and these IDs always seemed to be correct. However, other types of cast members could have scripts as well, and since they wouldn’t have this field, this solution wouldn’t work for them. Or so it seemed.

Long story short, we were treating too many bytes as type-specific data, and the scriptId was actually in the standard cast member info. Once that was fixed, every cast member had a single, correct scriptId associated with it. Use that to link cast members to the scripts, and no more duplicate scripts!

Grammar

Next was improvements to the Lingo grammar.

First, I needed to differentiate between statements and expressions. An expression by itself, like 2 + 2, isn’t a valid Lingo script - it needs to be an argument to a statement, like put 2 + 2. However, we were treating expressions and statements exactly the same, which allowed incorrect scripts and significantly complicated the grammar. Once this was fixed, half of the grammar’s 441 conflicts were gone.

Next, I needed to get rid of the differentiation between Lingo’s subroutine types. Confusingly, Lingo has (at least) 3 different types, with overlapping purposes:

  • Commands - These are built-in, and invoked by a call statement, like foo() or foo.
  • Functions - These are also built-in, and invoked by a call expression, like put foo(). Very rarely, you can also invoke them as statements.
  • Handlers - These are user-defined, and can be invoked by either call statements or call expressions.

Now, these are separate things, but they should only be treated separately during execution. Previously we were differentiating them in the grammar, which again complicated things.

With that done, I began general cleanup. Reorganizing things where conflicts could be eliminated, reducing the use of right recursion, and adding support for fun statements like this one:

put cast cast

What should this do? Why, of course, it prints the cast member whose ID is equal to the variable cast:

set cast = 1
put cast cast
-- (cast 1)

All in all, the grammar is now truer to the original, and we’re down to 6 conflicts from 441!

Lots of chop suey

Nathanael Gentry, 21/07/2020 | Source: ScummVM on Nathanael Gentry

Hey there!

My big showcase for the week’s work is a few scenes from the kids’ game Chop Suey, one of our primary Director 4 test cases:

Just a few weeks ago, Chop Suey ran at an almost unplayable crawl in ScummVM. Part of this was its reliance on Matte inks, for which I implemented a simple surface caching scheme and shaved about 20% off our buildbot’s target test time for Spaceship Warlock. This also made Chop Suey run much faster, though it still consumed CPU

Last week I called Chop Suey a Lingo-heavy game, and I was referring to how much it controls animation via puppets and the updateStage command. Because, as you saw, its cursors are bitmaps and certainly do not fit in the standard 16x16 Macintosh cursor box, Chop Suey introduces its own mouse update code and calls for the stage to be updated several dozen times each frame. Most of the inefficiencies were here.

I spent the early part of the week in much trial-and-error, working out the pieces of the renderer that were most inefficient under such repeated application. The idea is to do a little bit of work up front – checking flags and so forth – so the expense of redrawing a region of the screen is saved. (As I have realized, even when working on Chop Suey, very subtle bugs can arise from forgetting to check a rendering flag.) Even at usual framerates without much Lingo that doesn’t matter very much, but Lingo-heavy games like Chop Suey have shown dramatic improvement.

I also implemented another feature that is notable in Chop Suey by its absence. I said that Chop Suey does its own cursor handling. Well, the window manager was still drawing a regular cursor atop the bitmap, which looked pretty ugly. Near the end of the week, though, I added a flexible cursor class for the three cursor types Director can use: built-in cursors, cast (bitmap) cursors, and resource cursors. The last type is important for Majestic: Alien Encounter, but I haven’t seen custom cursors used much elsewhere. It was fun to implement, though. As a nice byproduct in Chop Suey, the default cursor is now properly turned off.

Oh, and I also spent most of a day trying to discover why some textboxes in Spaceship Warlock weren’t rendering properly, along with some other nettling MacGUI issues. The issue actually lay in the cast loading code, which I hadn’t touched much, but it’s always satisfying to squash a bug and learn more about the codebase in the process – even if your “fix” breaks other stuff. :)