Welcome!

Google Summer of Code 2022 has now ended, so this post will wrap up everything that’s been done and what is left to do.

The branch of my repository with the latest code for The Immortal can be found here: https://github.com/Quote58/scummvm/tree/immortal/engines/immortal

What has been done

– Initial analysis of the source code and ScummVM engine skeleton [relevant post] [relevant commit]

The original plan was to continue the implementation started in a previous GSoC for the DOS version of the game, however after taking a look through the source code for each version and discussing it with my mentor, the plan changed and I began work on implementing the Apple IIGS version instead.

– Support for ProDOS with Common::Archive [relevant post] [relevant commit]

The first challenge that presented itself when I began work on the Apple IIGS version was the file format for the game itself. The Immortal was originally packaged as ProDOS disks, which was an issue because ScummVM did not have support for reading this file format at all. Adding support required learning about and understanding the way ProDOS files work,  and experimenting until I had code that could parse the entire disk and list the files within. Once it was possible to get and return files from the disk with a file name, I was encouraged to integrate this with the Common::Archive class in ScummVM, and proceeded to do so.

– Compression Routines [relevant post] [relevant commit]

Once the file could be read by the engine, I decided to begin by tackling something that looked fairly self contained. The compression/decompression routines did not seem to be tied in to any other functions within the engine, and so could serve as a way to get comfortable with what I would be doing in the rest of the engine, through something with a clear purpose and input/output. In the linked post, I talk about the process of beginning to translate the assembly of the engine into C++, with the compression routines as the example.

– Initial engine and subsystem skeletons [relevant post] [relevant commit]

After the compression functions were written, the next step was to start analyzing the files in the engine and begin to understand the main path of the game logic, the connections between different files, and ultimately create the initial engine skeletons. This meant creating the outline of immortal.cpp, kernal.cpp, and logic.cpp, and creating the necessary function stubs for everything along the way.

– Palette processing [relevant post] [relevant commit]

While creating the skeleton for kernal.cpp, I decided to implement the main palette processing functions. There are not too many functions related to this, and it was a core component of the engine, so I felt it was good to implement along the way. It also served as an example of how the assembly code makes heavy use of bit manipulation and interesting math/programming principles.

– Scope and engine outline [relevant post]

One part of translating the engine that took some time to get sorted out, was the general structure of the engine itself. It isn’t initially clear how each component of the engine relates to each other, but it is important for knowing how to structure the translation, what data types to choose, how to manage scope, etc. The linked post goes over the entire process of sorting through the files and determining the general scope and relation of each component in terms of ‘layers’ in the engine. This was important for the translation going forward, as it could inform decisions about how each new component that was translated related to the whole engine.

– Story subsystem [relevant post] [relevant commit]

The ‘story’ subsystem in the game is crucial, as it defines many of the important aspects of each level of the game. It defines what and where the objects, enemies, and more are in each level, as well as the dialog strings, the sprite animations, and more. The way it is implemented in the source code is unusual, and the linked post goes into detail about it.

– Cycles [relevant post] [relevant commit]

‘Cycles’ are the way that this engine handles sprite animations, and the linked post goes over the oddities with them.

– Flameset [relevant commit]

‘Flamesets’ are the torches that you find on the walls in each level. They are not objects or enemies, but are in fact their own little subsystem consisting of a sprite cycle, and code for interacting with fireballs, and even dimming the screen when they are ‘off’.

– Sprite rendering [relevant post] [relevant commit]

Sprite rendering involved many different subsystems and took some time. This is because the method for preparing and drawing sprites is a bit unusual, and translating it accurately was a challenge. The post goes into detail about some aspects, but there are several others throughout which detail the process of understanding them.

– Text parsing and printing [relevant post] [relevant commit]

– Hit gauge [relevant commit]

This acts as the wizard’s health bar in the game, and is drawn with sprites from the font sprite sheet.

– Implemented many functions for Kernal, Logic, Level, Room, Sprites, and more, creating the skeleton of the engine in a way that supports a continued, accurate translation

What is left to do

I had hoped to have some of the map drawing in time for this post, but unfortunately the map data is stored in multiple file types (.CNM and .UNV) which require unpacking and processing before they are usable, and the formats are specific to the Apple IIGS version of the game. I am working on it, but it will take more time to decipher.

• Drawing level data from maze files with drawchr (requires deciphering the .CNM and .UNV files)
• Implementing an input layer through ScummVM (the engine code currently references ‘actions’ and has function stubs that are called for input, but functions for getting input from ScummVM need to be written)
• Object and Monster subsystems
• Music and sound implementation
• Filling out the rest of the static and dynamic story entries (the first level is written out and the structure is there, it just needs the data for the other levels to be written out) (the static data for Str and Cyc are included, but Motive, Program, and ObjType still need to be written out)
• Filling out remaining functions in kernal.cpp, logic.cpp, level.cpp, and room.cpp
• Implementing motives, pickups, and other object/enemy structures and their related functions

Conclusion

My time with Google Summer of Code 2022 has been wonderful, and in some ways I am sad for it to be over. However I am also grateful to have been a part of it this year, as I have learned a lot and have gotten to contribute to something I think is meaningful. ScummVM is a great program, and a great community.

I was hesitant at first about making these blog posts, but I ended up enjoying and learning from them. It allowed me to strengthen my understanding of the code and concepts at hand, and it served as a good organizational tool, allowing me to keep track of progress each week.

I will continue to work on The Immortal, building from I have done the past few months towards a complete implementation of the game. And if possible I will continue to update this blog with progress.

Lastly, I want to say Thank you. To Google and ScummVM for this opportunity, to my mentor Criezy for the advice and code reviews, to Sev and all the other ScummVM members that helped me (especially with Git!), to my brother for pushing me to apply for GSoC in the first place as well as his support and help along the way, and to anyone that has read any or all of these blog posts or simply followed the progress in any number of ways. I am truly grateful to all of you for this past summer.

Welcome

As shown above, text rendering is now implemented enough to display the introduction to the game. It’s kind of neat, because every time I had tested the game ever since implementing the logic system, it was trying to display this introduction sequence. It was just calling a function that couldn’t render text until now. Similarly, after these text sequences, it loads the first level from story, getting the basics of the room set up, but it can’t display anything until I implement the tile drawing (and some ‘universe’ properties for getting relative positions and whatnot).

There are still a few things to do for text rendering, but it can even handle ‘formatted’ text, which means the string has denoted at the start that it will use automatic page breaks and spacing. For example:

For this kind of text sequence, the game will actually check the distance before drawing a word, to see if the word will fit on the current line. If not, it will start a new line, and if there is no room there either it will clear the screen and start at the top. Very neat stuff.

The process of making text render is surprisingly complicated, but not for the reason you might think. The actual text parsing is fairly straightforward. It is the individual letter drawing that gets complex. To understand why, I need to mention first what’s going on with the text drawing at all.

For any given text, the game ‘writes’ it out to the screen. Whether it is the intro text, where it fades in to a fully rendered text screen, or it’s a dialog sequence that draws across the screen at a certain speed. In all cases, the same system is used. To draw a letter, whether it’s part of a string, or just a character on its own (ex. the password ‘certificate’, or the health bar), the game uses a virtual ‘Pen’ to do it. By which I mean, the source specifically calls it PenX and PenY. I think this refers to two things. 1. The coordinates are relative, and 2. Each letter is drawn individually, in a natural way. The first is pretty simple, ‘Pen’ coordinates are relative to the edges of the viewport. So any coordinates you give to the character drawing function will automatically assume the edges of the screen are included. The second, refers to both the fact that each letter drawn gets a potential delay (which can be influenced by what buttons are held or not held down), and that letters are drawn relative to their own size, and the size of the other letters. For example, you can see that the letter ‘i’ is much thinner than ‘m’, but it fits snugly between other letters as if they were a uniform size. The thing is, when you put together all of the sprite offsets and pen movements, there is a ton going on to draw a single pixel of a letter. The process looks something like this:

Starting with a Pen of (0,0), a text sequence starts by clearing the screen (drawing a black rectangle at the viewport), and then setting the penX to 8 and penY to 4. This is just to give any text a bit of space. Keep in mind that the screen edges get added on when the Pen becomes coordinates for the sprite drawing function.

After this, we start the string. The first 4 characters of the intro string are spaces, which simply add 8 pixels to the PenX position.

Next, the letter is a capital E, and all capitals get 8 pixels of extra space added to the X. This is because every letter will add 8 pixels to the PenX after they have drawn, to cover the basic width of a letter. However wide letters like m and w add an additional 8, and this applies again if the letter is a capital, as they are twice as large in this font. Like I mentioned before however, some letters are smaller, and actually subtract a number of pixels from the Pen. For example ‘t’ subtracts 3 pixels to make sure it is drawn closer to the last letter. This is all taken into account in the sprite sizes, but we’ll see that in a moment.

Now we are about to draw the sprite, but we’re in the middle of where we know the letter gets drawn, why is that? Well along with the Pen and screen parameters, the sprites have their own individual offsets and spacing. For letters, they have a centre X of 16 pixels. This means that before drawing, the sprite game will subtract 16 pixels from whatever X position it is at.

This leaves us pretty far away from where we know the sprite draws. But before we solve that, the next step is turning the X and Y coordinates into a pixel index (the original game used a mix of byte and pixel positions, and had to convert between them. This is because the virtual screen buffer where sprites draw contains two pixels per byte, for more info on that see the post just before this one. For ScummVM, the equivalent screen buffer is stored as one pixel per byte). After this, we can start looking at the sprite data itself, which begins with a scan line offset of 20 pixels. This offset puts our index right where it needs to be to draw the first two pixels. Following that, each scan line defines the difference in position to get to the first pixel of the next line.

Putting this all together, we can see that there are many steps to translate a Pen position, to the actual sprite being drawn.

But it ensures that text renders like that, and not like this:

Next

This is now the final week (19 – 26), so this week will be both working on the next code (which I’ll mention in a moment), and also cleaning up the code a bit, getting the commits uploaded, and then making the final GSOC blog post for the submission. The code that I will try to get working this week is for drawing the map tiles (which does not use the sprite function at all, contrary to what I thought before due to sprite properties of some kind being referenced there). This first requires loading in the ‘maze’ data, and then implementing the drawing routines for tiles. These are quite complex, and ultimately I may need to just write a function for now that interprets the data on its own, and fully implement the source process in the future. Either way, I will do what I can in the time remaining. Thanks for reading!

Hello!

This is a bit of a special post because I finally have stuff to show that actually has a visual component!

Getting to this point took quite a few steps, as you can see below. But there are many interesting things to take away from it which I will get into later in this post.

For now however, I’ll start by mentioning what has been done and where things are at currently.

Sprite drawing is complicated (if the past several posts had not given this impression already). Sprite drawing, the way the source does it, is very complicated. More important however, is that sprite drawing is also different on the Apple IIGS compared to ScummVM. The result is that already complicated things get tied up in other factors, and make it hard to parse any of it at all. I’ll get into detail about it momentarily, but suffice it to say that my implementation is as close as I think I can get to the same logic for the moment. It’s not exactly the same methodology, but the exact methodology is based on certain things that aren’t relevant for the port. This is all to say that sprite drawing in general is implemented. You can call superSprites() with a sprite and a frame for the sprite, and it will draw that sprite on the screen. This doesn’t just mean finding the pixel data and making it fit a generic new function though, this implementation goes through the same steps the source does, which means it also clips the sprites correctly, and utilizes the BMW and not a hardcoded screen width, etc. It even includes the check for whether the Y position has changed, which was an optimization designed to avoid excess multiplication when drawing things like letters, since they will generally be the same Y position for several in a row. Making it work required implementing superSprites(), clipSprite(), and spriteAligned(). And sorting out the methodology was quite a process. As well, now that superSprites() was working, I was able to uncomment the call inside printChr() and to my genuine delight, the function worked and printed letters just like it is supposed to. This is a big deal because that was one of the functions that could only really be tested with a visual component, and until getting sprite drawing working, there was no easy way to tell if it was working correctly, so I assumed there would be small bugs to fix right away. Along the same lines, the hit gauge (this game’s heads up display essentially), also worked right away. These were a huge relief, as it means at least a couple of the building blocks I’ve been working on are in fact working.

Font rendering

Hit gauge rendering:

Now for the nitty gritty part. What even is the pipeline for rendering in this game anyway? We’ve already established a few things, so I’ll start there:
Bitmap -> Frame -> DataSprite -> Cycle -> Sprite -> Priority Sorting -> Munge -> Screen
The last three are not relevant to this discussion, and Bitmap as I mentioned in a previous post, is more complicated, so I’ll adjust for those:
Scanline Bitmap -> Scanline offset -> Frame -> DataSprite -> Cycle -> Sprite

However, it gets more detailed than that. Not only is the data structure for the bitmap actually a set of smaller data structures, but the way those pixels are applied to the screen is itself a multi-step procedure. This is due largely to transparency, and the difference between console and Apple IIGS screen rendering. So first off, here is a more accurate pipeline:
Screen pixel -> Transparency mask -> Bitmap pixel -> Pixel drawing function -> Scanline Bitmap -> Scanline Offset -> Frame -> DataSprite -> Cycle -> Sprite

Now, why did I include a function as part of the structural pipeline? Well it’s all part of the pixel drawing system. Before understanding this system however, we need to go over another difference between consoles and PCs. On a console (depending on the console I’m sure), you tell the hardware where to draw a sprite, and the layer priorities, and the hardware handles taking all the graphics data that is at that location, and combining the pixels based on their priority and transparency, into a single pixel for each point. On the PC however, that step was up to the developer. The virtual screen buffer, which is what will be used for drawing to the actual screen on the Apple IIGS, can only be manipulated in bytes, nothing smaller. However, the pixel data of the resources are stored in smaller quantities. The pixel depth is 4bpp (Bits Per Pixel), which means each pixel is contained in one nyble (half of one byte). However since the smallest type of data we can load and store on the 65816 is a single byte, we can not just load from the bitmap and store directly to the screen buffer. We need to at least mask half of the bits for one pixel, and then half for the next, and finally add the two together. This brings up other questions, like alignment, but I’m only going to discuss the main one, transparency. In The Immortal, transparency is defined by a colour value of 0. So you may think that the process would be to extract one pixel of data, check if it’s 0, and if not then add it to a temporary byte, do the same for the other, and then combine the screen byte with the final byte. But that would be a little bit costly. Not much, but considering we are trying to do in software what a console would do in hardware, we need it to be extremely fast. After all, whatever we do will happen for every single pixel of the sprite.

This is where the method used in this game comes in. Instead of treating the pixel data byte as two individual pixels, The Immortal treats the two pixels as one value. And this value, is used as an index into a special array. The game generates an array that contains bit masks which cover a) all bits, b) only the low byte, c) only the high byte, or d) no bits at all. This is because what it wants to determine is what nyble, if any, needs to be empty due to the colour value being transparent. So, if one nyble of the pixel data is 0, then we want to get a bitmask that has 0 where that nyble is. What this ends up looking like is FF, F0…, (0F, 00…)…

If the pixel data is 00, that means both pixels are transparent, and therefor we want all the bits of the screen data, which we get with FF. At 01 – 0F, we want to keep the bits of the first screen pixel, but not the second. At 10, we only want the second nyble, etc. This goes on all the way until FF, providing a mask for every possible combination of colours those pixels can produce.

This mask is then used in between the screen pixels and the bitmap pixels, to combine them into a single byte. (ScreenPixels & PixelMask) | BitmapPixels
This takes up a ton of memory, but it is extremely fast. It handles two pixels at once, and only has to index an array, rather than use any conditionals. It also handles transparency at the same time as applying the pixels to the screen.

There is another thing that the sprite system does which is fascinating, but since similar things are done in the drawChr subsystem (the one for drawing tiles on the screen as opposed to sprites), I’m sure I will cover that next time.

What’s next

As for what’s next? A number of things very soon. With these remaining two weeks, I hope to implement text parsing so that the intro to the game can be shown, as well as tile rendering, so that I can potentially show the first room of the game. There are many steps in between, so we will see how it goes. But at least now I can provide a visual component from the game itself for some of the things that I talk about in these blog posts!

Thank you all for reading, and for now I will leave you with a gif which shows that the sprites are getting loaded in correctly:

My name is Emanuele Grisenti (aka grisenti) and in the past three months I’ve been working on integrating the HPL1 engine, made by Frictional Games, in ScummVM, with the goal of supporting the 2007 game Penumbra: Overture, created by the same company. The source code for both the engine and the game were made publicly available in 2010, and my job was to replace the engine’s low-level feature (input, sound and graphics), with ScummVM’s equivalents. This proved much harder than I initially thought, especially working with the physics library, for which a different version had to be used, since the source code for the version used in the original engine is not available.  

The code can be found at https://github.com/grisenti/scummvm/commits/hpl1-gsoc, with the first commit being 3727a5caf8e821982b0e0008a3bda0da8c11f238. 

What’s been done

Most of the goals outlined in the initial proposal have been met, and the game can almost be played to the end, but there are still serious issues in final levels.

What’s left to do

• Resolve the game breaking bugs left at the end of the game.
• Removing the use of stl containers, which was a goal of the initial proposal but unfortunately had to be put aside to work on other problems.  
• Improving portability by removing platform specific code and other non-portable functionality.
  
• Implementing a software renderer to make the game playable even without OpenGL. 
• Cleaning up most of the code, including compilation warnings, unused functionality and stylistic changes.

Closing thoughts

Overall, it was an amazing experience and a fantastic learning opportunity, for which I want to thank anyone who gave their time to help out or to read the discussions on discord or the weekly blogs. It’s going to be a while before the game will appear in a future ScummVM release, but I definitely want to keep working to get it there.

Hi! I am Avijeet Maurya, one of the GSoC 2022 contributors working on ScummVM. The main focus of my GSoC project was on improving the condition of several existing game engines through bug fixes and implementing some other improvements.

What was done

GLK Scott

Implemented support for C64 and TI99/4A games by porting the GLK Scott interpreter from spatterlight.

Blog Posts:

Week 1 – Getting started.

Week 2 – Some progress and cleanup

Week 3 – Community Bonding period ends

Week 4 – Start of the coding period

Week 5 – Finishing up Scott

Pull requests: #1, #2, #3, #4

PINK

Various bug fixes related to the in-game PDA and updating the code to work with the newer MacGUI implementation. Also fixed bugs related to Hebrew support.

Blog Posts:

Week 6 – Working on PINK

Week 7 – Finishing up PINK

Week 11 – Finishing WAGE and PINK

Pull requests: #1, #2, #3, #4, #5, #6, #7, #8

WAGE

Various bug fixes and MacGUI related improvements.

Blog Posts:

Week 8 – Working on WAGE

Week 9 – Working on WAGE II

Week 10 – Working on WAGE III

Week 11 – Finishing WAGE and PINK

Pull Requests: #1

MacVenture

Various bug fixes related to GUI.

Blog Posts:

Week 12 – Working on MacVenture

Week 13 – Working on MacVenture II

Week 14 – Working on MacVenture III

Week 15 – Wrapping up

Pull requests: #1

Avalanche

Fixed two gameplay bugs.

Blog Posts:

Week 15 Part II – Working on Avalanche

Pull requests: #1

What is Left

The leftover work is distributed between MacVenture and Avalanche. For MacVenture I still need to:

• Implement lasso selection
• Add IIGS support

For Avalanche I need to work on the TODOs listed on the wiki page.

Closing Thoughts

GSoC wasn’t something I could have completed on my own so I would like to start by thanking the ScummVM team for being so helpful at all stages of the program. Special thanks to sev, eientei, Jaderlund and Voltya for helping out with different parts of the project.

Getting started with contributing to other open source projects was always a daunting prospect because of how difficult it was to actually understand the huge codebase of various projects but this time I managed to overcome that through a combination of just going at the problem at hand and also asking for help when needed.

A really important thing I discovered through GSoC was to have a positive mindset while coding. I never really felt nervous, even when I was stuck on something because I knew I had people to rely on and ask questions to. This helped me progress through in a much more relaxed manner which is something I don’t think I could have done before.

And speaking of asking questions, it was something I always hesitated to do because I felt like it was a stupid thing to do but now I know that it’s better to admit that you don’t understand something rather than pretend that you do and cause problems later.

Another thing I have learned through GSoC is getting into the habit of working daily. I used to be very irregular in when I work and I am happy to have changed that.

I also liked writing the weekly blogs and from now on I’ll try to write about what I work on, even if it’s just some personal notes. You need to really understand what you are working on to write about it and I often had some new insights about the problems I was facing when I tried to describe them in my blog.

And in the end I want to thank all the people who followed me through this journey and read the blog posts. I really enjoyed writing them and it’s great knowing that there were people who interested in reading what I had to say!

Thanks for reading!

A small addition to Week 15’s entry is some work I did on Avalanche. There’s wasn’t much time left when I got to Avalanche so after looking at the pending issues, I decided to tackle some gameplay bugs as those are generally simpler than implementing entirely new features.

The first one was about the in-game movement direction indicator.

The game has two different movement methods, arrow keys and clicking. The bugged behavior would update the direction indicator only when arrow keys were used. To fix this was quite simple, I just found the function used to handle keyboard input and checked how the direction indicator was being updated for the corresponding arrow key input. I then went to the part of the code which handles mouse input and used those same function and now it works perfectly.

The second bug was something I found while playing through the game. Spoilers for the game upcoming incase that’s something you are worried about.

There’s a segment where you have to climb up a dais but it’s rigged to shoot a arrow as soon as you do that. The solution to progress is to quickly climb down to avoid the arrow. However, the bugged behavior would insta-kill you as soon as you entered the command to climb up.

To fix this I looked for the code which handles the arrow firing event. Luckily the error in that code was quite obvious.

The sprite data is what holds the coordinates of the sprite. Since the same sprite data was being used, they would immediately pass the collision check. I replaced this to use the correct sprite data for the main character.

This was the last fix I could make for Avalanche in the time I had and also the last of my contribution for this GSoC. This was a great journey and working on ScummVM was a good experience for me. I’ll be posting a GSoC summary post after this (and you might have already seen a password protected version thanks to a mess up) so look forward to that. Thanks for sticking to the end with me!

This blog post is the culmination of my GSoC 2022 journey.
This summer, I worked on the ScummVM Director engine, where I completed the STUB code in the engine and worked on fixing various issues in it and bringing it more in line with the original working, thus improving the compatibility of games (D4 and older).

This is a list of the Pull Requests I created during the summer in ScummVM. They completely describe each issue they are supposed to fix/implement. My initial work was with the STUBs. Here is a list of STUBs that were resolved.

Next, I worked on the various issues encountered in certain targets of Director Engine which get updated on this Trello board. This board also contains the open issues of the Director engine.

I also created a couple of test movies to gain a better understanding of the internals of the original Macromedia Director. They can be found here.

It was a great experience working with the ScummVM Director devteam (sev, djsrv, rvanlaar, mstea, moralrecordings and all the other devs). I consider myself privileged to have contributed to such a popular and well-maintained open source project. I learnt a lot of things this summer from this awesome experience.

Since director is an ongoing project and the plan is to support games till D7, I will be sticking around and contributing to the Director engine (and ScummVM) after GSoC. I would love to continue being a part of the community and volunteer in the project.

A huge thanks to my mentors sev and djsrv (and the devteam) to help me out throughout the duration of this programme and answering even my dumbest queries with patience 😀

This is the concluding post of my GSoC journey. I loved the project and the people and would continue to be a part of it. Thank you everyone 😀

This week, I mostly worked on the PopUpMenuXObj. sev had outlined to me the steps that would be needed to get it working.

The first step was to make the MacPopUp menu constructor. Now MacPopUp is very similar to MacMenu (essentially, it is a menu with a single submenu. It can have BITD images as menuItems, and can be drawn anywhere on the screen instead of just the top)

So MacPopUp is derived from MacMenu. We have MacWindowManager::addMenu(), which removes the current menu if one exists. creates a new one, sets it as the menu of the movie and adds that to the _windows array. To change this for MacPopUp, we need to remove the part where it is set as the menu of the movie, and we assign the index of the menu in the _windows array ourselves. (This is something done in the original director engine too to prevent collision of popupmenu IDs with the IDs of other windows). So the MacWindowManager::addPopUpMenu() has been made to do this (creates a MacPopUp instead of a MacMenu in our case)

Then a null item is added to the menu, and the submenu is created by the provided string. These functions take care of creation of MacPopUp, and the menu object is created using this.

Now, for drawing, I had to change some functions of MacMenu. Most of the changes were in MacMenu::mouseClick(), MacMenu::draw() and MacMenu::calcDimensions.

The changes were mainly the passing of drawing coordinates as a parameter in calcDimensions, adding a forceDraw flag in mouseClick to allow drawing anywhere on the screen, and removal of drawing of top bar in draw().

The end result is this:
Now, the changes to these functions were done directly in MacMenu. Adding a macpopupmenu.cpp to the project is resulting in LNK2019 error. I had tried resolving it before writing the constructor, but then went ahead with just the header. I will try to fix it again now and override them in the MacPopUp class.

I will finish this and then pick up the issue I have not touched since long – the window options. Some of the work on that is already done. I will have to finish up the remaining.

This is the 12th weekly update of my GSoC program. I will be submitting my work before 12 September. My experience with ScummVM has been amazing this summer and I would be sticking around on the discord and continue contributing to the project (to the Director engine and other stuff – like there was a content verification project proposed in the GSoC ideas list which I would definitely like to work on, not as GSoC project, but voluntary contribution). The experience with my mentors has been great, both sev and djsrv were quite helpful and understanding throughout the time, and were more than happy to help me anytime (This is the same for the rest of the director engine team – rvanlaar, mstea and moralrecordings). I am happy to have contributed to the director engine and would continue doing that.

See you! 😀

Welcome

Today isn’t a weekly update, since I did one a few days ago, instead I wanted to discuss something specifically to do with moving between Assembly and C. So, in a smaller, shorter capacity, this is a follow up to the previous post about translating assembly code.

I was recently implementing a function for the utilities file called ‘inside’. This function is generally pretty simple, just checking if a given point is within a given distance of another point. There are many ways one could implement such a function, but the one chosen by The Immortal makes use of a fairly complicated bit of branching if you aren’t used to it. It branches almost entirely on the carry flag, and because it is all within a fairly small space, it can branch in and out of nested statements with ease. The complication is what it brings up for the concept of control flow, and how Assembly differs from modern programming languages. The basic idea here is that any program has a logical flow of information that is dictated by ‘control’ statements. These statements, labels, and structures are like rocks in a river. The water is always flowing, and the rocks in the path change the direction of that flow, or even prevent it from moving further. Of course there are also jump statements, which are more like the water falling into a container and the container being carried somewhere else. Let’s leave the analogy there before it gets harder to track… Jump statements in Assembly are exactly what they sound like, they jump to another position in memory. In higher level languages however the equivalent would likely be stack frames. Or rather,  jump statements are reinterpreted in higher level languages to be larger order control structures, rather than simple address A to address B movement. However what I want to focus on are branch statements, because these are more complicated to translate directly. The structure of a subroutine in Assembly is pretty straightforward to translate into a function in C, because they accomplish more or less the same thing and are used in similar ways. However translating a system of branches can become much more complicated than it may seem at first glance. Part of this is because it is not always clear what the direct translation would be, but also because the compiler will often change the structure to something that is more efficient in Assembly after you’ve written it regardless. The function I mentioned earlier is what got me thinking about branches and interpretations recently, and I think there is something very interesting to talk about here. So without further ado, let’s start by taking a look at that function in Assembly:
(I have converted the labels to generic ‘+’ labels, and changed certain names to remove anything non-generic from the source. Also of note, ‘add’ is a macro for clc : adc)

Okay, so here we can see many different branch statements. BEQ, BCS, and BCC are what dictate the flow of information in this routine, and they branch on the equal flag (Branch if EQual) and the carry flag (Branch on Carry Set, and Branch on Carry Clear). The specifics of how they branch is not super important right now though, because what we are going to do first is start to abstract the structure out so that it can be translated. The specific branch statements can not be directly translated (or at least should not be, as you do technically have a goto statement in C), so the first step is to interpret the structure in the abstract, so that it can be converted to one of the supported structures of a higher level language. It’s a little bit like a math equation that is in the wrong form to be used right now. Most of the time this step is done in your head as you read it, but I wanted to visualize the steps because I think the process has something to say about programming generally.

To start, we can identify the control statements and then group individual statements by their relation to other statements, not necessarily in the same physical order as the code is written. For example, many different statements branch to ‘end’, so we need to put that somewhere that we can connect to the others. The result is a sort of three column wide structure.

Now, what can be gleaned from this diagram? I think the most important thing to take away from it, is that many statements connect to the Carry Set exit, and only two statements connect to Carry Clear. This makes sense, as the routine needs to determine whether the point is within a given distance on any side of the point. There are two sides that are before the point, and two sides after. So having a CMP dist and CMP negDist, while all other statements have the other exit, tracks. The other important thing here, is that with the statements grouped, we no longer need the specifics of their Assembly code. We can now abstract them out to their identifiers, to make it easier to find the right control structure. The control structure in C will be abstract, so we need to make sure the one we are deciphering from this Assembly is on the same level of abstraction. So from here, we can simplify the diagram by allowing there to be multiple copies of that exit statement that so many statements connect to. The logic is the same, if A can go to J, and D can go to J, but J can not go anywhere, then the statement A -> J <- D is the same as A -> J & D -> J. This will make it much easier to find a control structure, and is something that I will come back to in the end. To be clear though, you don’t have to have multiple copies of J. In fact originally I was stubbornly writing code that preserved this aspect (I even had a great pun lined up for the post, about ‘!(E) code’, where E was a particularly convoluted statement). However as we will see, the abstract version with multiple J statements will translate much more quickly. Here’s what we get when we simplify the previous diagram:

I did one other thing with this diagram, which was to highlight the two groups of statements. This is because now that the diagram is easier to read, we can see that there are simply two distinct groups of branching structures, each containing the same layout. 1 -> 2 & 3, 2 -> Carry Clear/Carry Set & 3 -> Carry Clear/Carry Set. This is crucial, because now we can start to look at this in terms of abstract control structures in C. We don’t have to care whether those statements use BCS or BEQ, those are semantics that can be replicated in C once we know the structure. What we can now latch on to, is that structure. So let’s redraw the diagram again, but this time with the C structures in mind:

And now, we finally have something that can be directly translated into C. We have control structures based on if-else statements, and there are no convoluted, criss crossing arrows. We can take that diagram, and use it to convert the original Assembly into C like this:

Now lastly, I want to come back to the point about having multiple J statements. As we can see from the C code, we have 4 instances of ‘return false’. This looks at first like it is not the direct translation of the Assembly, because it has multiple instances of the statement, not just references to the statement. However, what we need to keep in mind, is the level of abstraction. C code is not machine code. It can be a bit weird to think about, but where Assembly is interpreted more or less exactly as is by the assembler (the program that converts Assembly code into a string of binary), C code is not always interpreted exactly by the compiler (the program which creates Assembly code from C code). This is almost always a good thing, because the compiler is very smart and can usually find the most efficient way to write C code as Assembly. As a result, what needs to be translated is not the exact statements, it is the logic of those statements. The river can look a little different, as long as the water flows through the same path every time. On a macro level, this is actually what ScummVM does. It replaces exact machine code with logically equivalent C++ code. The result is that the game is preserved accurately (as accurately as possible without emulation of some kind), even though the code is not necessarily identical. This has a huge advantage, because just like we saw on a micro scale, preserving logic across levels of abstraction creates choice in the way it is brought back down to lower abstraction, without sacrificing the logic itself. On the scale of the code we looked at, this meant we had choice in the control structure we chose in C (for example, we could have used a switch statement instead of if-else). On a macro scale, this means the game itself can be converted to run on any modern system, despite the raw source code being anything but compatible with modern systems. I think it’s fascinating personally.

Okay, thanks for reading everyone. I’ll have a regular weekly update next week that will focus on the progress towards screen rendering. This was just something I had been thinking about while translating various functions.