May 26, 2016 06:00 PM All times are UTC.
Welcome to the ScummVM planet - This aggregates the personal blogs of developers, teams members and active participants from all around the ScummVM community. If you wish to subscribe to updates to the planet or individual blogs please use the links on the right hand side. To add your blog to the planet contact DJWillis.
This week, I’m working on extracting the components of the WAGE engine that are reusable to other engines (such as… you guessed it, MacVenture), specifically that old and lovely black and white Macintosh GUI.
This, in addition to easing the burden of having to create these components myself, has served to give me insight about how an engine is structured. After this, I have a much clearer idea of how I should implement MacVenture. It mught be worth to revisit the order in which I planned to write the components.
However, as with every time that you touch other people’s code, there is a constant fear that you might have broken something, even more so if we take into account the lack of tests for engines. However, little by little, I think that I arrived to a working solution.
The Constant Struggle for Style (that’s a pun)
As it turns out, not all Macintosh GUIs were created equal, and thus they need different style constants (hence the pun) to adjust fun things like border width, border color, border pattern, window border, window frame border… well, you get the idea.
For comparison’s sake, here are two screensoths. One is from a WAGE game (Another Fine Mess), and the other is from a MacVenture game (Uninvited):
Since the styles are so different, for now at least every engine will hold to it’s own constants, and they will be isolated from the rest. If any of the common files (macwindow.cpp, I’m looking at you) would need any constants, these will be defined in the implementation, and not in the interface, so as to keep things nice and tidy. For me in particular, this has several ramifications:
Obviously, I’ll have to tinker my fair share with the constants when I’m implementing MacVenture.
If and when I implement the Apple II graphics, I will have to do it inside MacVenture also, or spend a very sizeable portion of the schedule making the system abstract enough to handle it.
The rest of the week
“But it’s still just Thursday”, you say. “What are you going to do with the rest of the week?”
Well, my goal is to submit the code I already have in my fork to review by my mentor and anybody else that wants to see it, and then get it ready to issue a PR. There are still a couple of things I’m unclear on, I think I can make a general style work. We’ll see.
It has been a couple of months since ScummVM 1.8.0, which means it’s time for
a new point release! ScummVM 1.8.1 is now available with fixes for important
bugs and support for even more platforms.
One of the most important "features" of this release is a major upgrade
to the Android port. This upgrade also adds support for the OUYA console.
We are currently working on bringing this new release straight to Google Play,
but for the impatient among you, feel free to go ahead and
install the APKs manually. The Android
port now comes in two flavours, our "classic" OpenGL-based one, and a new
variant based on SDL. The SDL port has better compatibility with some Android devices
and also works better with some engines. Please give it a try and tell us on
our forums or via
the bug tracker about your experience.
We will also be adding Nintendo 3DS support! As of release time, the new binaries
have not yet been built by our new porter, but they should be available soon.
Keep your eyes peeled!
64-bit Mac binaries now include experimental automatic update checking using
Sparkle. Auto-updating is selected as an
opt-in during your first start up, and by default checks weekly for new releases.
When version 1.9.0 is released, users of 1.8.1 will be presented with
a nice dialog for updating. If you'd like to see a similar feature for Windows or
other platforms, let us know! All the details on how the update process works can be
No ScummVM release would be complete without game engine
updates! ScummVM 1.8.1 includes fixes for Drascula, Legend of Kyrandia,
Labyrinth of Time, localized versions of I Have No Mouth,
Broken Sword 2.5 and improvements to the Windows, OS X and GCW0 ports.
So, the last time I wrote about doing some preparation work before GSoC starts and I believe I did quite good.
JSON parser works fine, even though we have to replace all unicode characters into ’?’ to make it work.
Now I’m checking that my code compiles not only with MSVC, but also with gcc through MinGW shell. Thus, I have tested the configure changes sev made, and libcurl detection and usage is successful.
There is a small «framework» for libcurl already: I’ve added ConnMan (ConnectionManager, similarly to ConfMan), which could be used to start curl requests. There is a special CurlJsonRequest, which reads the whole response, parses as JSON and then passes the result to its caller. Finally, I’ve commited Callbacks yesterday. I still keep those in a separate branch, because some edits might be necessary, but I believe it to be a very good thing to have.
These Callback classes are not just plain pointer to function stuff. It’s «object-oriented callbacks», meaning it’s actually «pointer to an object and its method». Plus, I also made it so one can specify an argument type. Yes, those are templates. I actually remembered, that I already did such thing a few years back while I was at school. «Functor» might be a wrong name to call it, but the idea is still the same: there is a base class, which doesn’t know anything about the class, whose method we’re going to point to, and there is a derived class, which implements it. We’re using pointers to the base class, so we don’t care whether it’s Callback<A> or Callback<B> or SomeOtherCallbackImplementation. Thus, we don’t have to write all these ugly global function callbacks, cast the only void * parameter to some struct and use fields of this struct to do the work. We just write Callback<ClassName> and pass ClassName::method, and so this method would be automatically called when this callback’s operator() is called. If we want to specify, that ClassName::method accepts not void *, but, for example, AnotherClass, we just write Callback<ClassName, AnotherClass> (note that it accepts not a pointer, but an object itself). It’s as simple as that!
I also wanted to do something about Storages and save corresponding access tokens and such into configuration file. Well, right now it remembers the access token of the «current» storage, and thus I’m not authenticating every time I launch ScummVM. Yet, I’d like to remember tokens of all connected storages, so user can easily switch between those.
Finally, there was a «writing API skeleton» in my plan. Well, this whole cloud system works as I thought it would. There are no real API method implementations, but apart from that it looks fine.
OK, so I’d have to start working next week. The general plan is to design API and implement Dropbox and some other provider support. My proposal schedule states the following:
May 23 — May 29 Write Storage interface, implement tokens saving.
Add Dropbox and OneDrive storages.
May 30 — June 5 Implement file downloading (Dropbox and OneDrive).
Add auto detection procedure running after the download.
Storage interface is more or less there, token is saved. «Add» there doesn’t imply storage would be completely functional, so adding Dropbox storage is also done.
So, the actual plan is to upgrade Storage to have some config saving related methods and make tokens saving feature support multiple Storages. After that, I guess I’d add OneDrive stub and start implementing some API methods in Dropbox and OneDrive backends.
I’m still having university studies here and exams are getting close. This means sometimes I’d have to study instead of work, and I might end up behind the schedule, not ahead of it. And that means I’m going to have no weekends closer to the midterm and after it =)
It's been a while since my last post... I could say, that I didn't have the time, but honestly, I had plenty, I'm just damn lazy. Also, I had a little hiccup, when I tried to implement a part of a display function. Started typing it up, realized that it has dependencies, so I went to check out those parts but then those parts had dependencies and so it went and I went with it down the rabbit hole. By the time a I realized that I wrote/imported about a thousand lines of code since the last commit and the last time I compiled it was about three days ago it was too late to go back (actually I could have with some additional work, I just thought it was no big deal). This will forever bare it's mark in dm's commit history. Long story short, I left out some details in some functions and did away with loose ends (otherwise I'd still be typing up code for that monsterous commit), and ran it. Little wonder, that it crashed with some bizarre results after a few init functions. Then I spent about two days fixing code, slowly stepping through it line by line, the result was that now file parsing, aside from some minor details is very much complete, there is still some untested code, but that music belongs to the future.
Still, it was a fun experience (who wouldn't want to have a date with the debugger at three o'clock in the morning), I also learned from my mistake and the next time I am to write some functionality into the engine, I'll list the dependencies first and start from the bottom up. That should help prevent accidents like this in the future.
Finally, here is a gif from the zeroth level of the dungeon:
PS: sev was kind enough to lend his time and wisdom in git wizardry, so the repo (sans scummvm code) with the commits has been moved here.
At the moment we are working with simple examples, where pictures constitute the bulk of casts. So I decided to start working on problems such loading score data (timeline structure) and addition of tempo channel. In the end we will get our first simple movie.
As the coding phase of GSOC approaches, a frantic wave of tasks and University projects has taken over my life. There’s a lot of work to do, and not a whole lot of time to do it (4 days for 4 projects). I am terrified of May and June, but I think that I’ll manage. After that, I’ll be able to dedicate as much time as I want (not just as much as I promised).
Meanwhile, I have updated the planning of the project, with an improved schedule to incorporate some of the changes that were necessary. As an exercise in transparency and pushing oneself, the full schedule will be openly available on Trello, for anyone interested to follow. Of course, one can also follow the commit history in my fork of ScummVM - it’s up to the reader.
For now, I leave you with a snapshot of the milestones of the project:
After discussing API design, I’ve done some research on Dropbox, Google Drive and OneDrive API and discussed further steps with Peter and Eugene. I’ve identified some preparation steps I should do before GSoC starts. Doing this preparation work would also get me familiar with ScummVM code, coding & formatting conventions and commit guidelines.
I’m working on it in my public fork of scummvm repo on Github right in the master branch. Eugene helped me with configure (which I’m not using yet, as I’ve started with MSVC here) and watches how I’m doing.
My preparation work plan includes the following tasks:
integrating JSON parser;
writing API skeleton;
adding libcurl and writing some simple wrapper for it;
adding some cloud-related configs keys, so ScummVM would remember my access token and I won’t have to authenticate every time I launch it.
We’ve decided that SimpleJSON is a good library to use as JSON parser in ScummVM. It’s really quite simple library, which consists of two classes and uses C++ standard library. It is now available as Common::JSON in my fork and uses ScummVM classes.
A few weeks back, I still was thinking that all cloud-related work would be done within separate thread spawned in main(). But when I’ve decided to actually add such thread (so my JSON examples could work and not delay ScummVM’s launcher), I understood that the original idea to use libcurl’s blocking functions is a bad idea. libcurl’s functions block thread execution until the request is complete. Because of that, our cloud thread won’t react to the user’s commands when it’s «busy». And that means if you started downloading 2 GB on 56 kbps, you can’t cancel it without killing ScummVM process!
That’s totally not what we want. There is no direct thread spawning in ScummVM and even though TimerManager actually runs callbacks in separate threads, such threads might work even when ScummVM main thread is finished (and TimerManager could be implemented so it won’t be running callbacks in separate threads anymore). So, I had to think of something. At first my idea was to add a callback through TimerManager, which would be executed every second. It would’ve checked whether the user wanted ScummVM to make some API requests and whether these requests were complete. But most of the time we don’t work with cloud, and thus such callback would be called even though it was unnecessary.
Now the idea is a bit more complex. We would have to use async libcurl functions, because we don’t want our callback to block. So, we can’t make our API methods to return the requested information, but instead those should receive some function pointer and call the passed function when this information is ready. I also believe that there could be more than one request pending, and some methods would require more than one REST API request.
Thus, I think of making special Request class, which would act as a base class for all methods implementations. For example, ListDirectoryRequest would be not only requesting the first «page» of directory contents, but make such requests until whole directory is listed. When all the information is gathered, it would call the callback.
Storage class, which would actually represent cloud storage API in ScummVM, has all the API methods we need. But these methods would be creating new Request objects only. The implementation would be hidden within these Request classes. So, Storage class would also contain some Request list. When it’s empty, there is nothing to do for Storage, and it does nothing. But when it is not, Storage would start a timer and poll these requests, whether they have complete what they were doing. When the request is complete, it is removed from this list. When list gets empty, Storage stops the timer.
Finally, there is also a CloudManager. This is a class, which would be loading all connected storages from ScummVM configs and providing fast access to the currently active Storage. So, the idea is to have all the storages loaded and ready, so user can easily switch between them in the Options menu. But only one Storage would be «active». This storage would be used to sync saves, upload and download files until user changes that. CloudManager would not only have a getter for it, but also a few «shortcut» methods, which would be using current storage within. For example, you won’t have to do cloudManager->getCurrentStorage()->syncSaves(). Instead, you can just do cloudManager->syncSaves().
I have not implemented Requests yet, but there are already some simple CloudManager and Storage stubs.
UPD: I’ve implemented first simple Request today. It’s not doing anything useful, but it shows that requests can «work» for a few handler() calls and then stop. When no requests are working, timer is automatically stopped.
Why do we need timer in the first place? Well, that’s because we want to know whether request is complete and we have to poll curl_multi_info_read to do that. Plus, this allows us to react to user’s commands. For example, user might want to cancel the download, and then Request would be stopped on the next timer tick.
Back to Java.
I recently got an assignment for one of my Artificial Intelligence courses: Given this set of basic classes (the ones in package “main” in the image), test the outcomes of three different algorithms, for the same problem. The problem was to “find the circle with the greatest radius that fits in a 1024x1024 board, and doesn’t overlap with any of these other circles that are already on the board”. We had to solve it using Brute Force, a Random solution generator, and a Genetic algorithm. Fair enough.
However, as you will note from the classes given to us, there is no way to generate the problems, or to visualize the solutions to extract data from them, so we had our work cut out for us. I will spare you the details, but where a simple logger would have sufficed, what I ended up doing was:
A problem generator, capable of generating an arbitrary number of problems and store the solution.
A simple GUI to visualize the outcome.
A logger, to record the genetic algirithm’s data.
A command line argument interpreter, to allow the user to generate problems, choose solvers, read problems from a file, or even just display a bunch of solutions from the command line.
So, 5 days of work and about 20 commits later, this is the result:
This evolution is heavily influenced by the YAGNI principle, wherein I don’t abstract anything until I need to, I refactor heavily, and I just focus on adding more functionality. I’m still a beginner in this kind of development (I was tought only waterfall), but I’m really enjoying it, and it makes work much, much easier and stress-free. I got the idea from this book, and I believe it’s a way better programming style.
So, after this little idea-flushing, I bid thee goodbye!
The engine stub has been created and minimal file parsing has been added, more specifically: the engine can now load and parse the graphics.dat file and decompress the images contained into bitmaps on demand, other than that not much more, but at least it's something visual. Also related to files, uncompressing the Dungeon.dat is now possible, but it's contents have not been used up yet. You can check the commits on github if you like, feel free to comment, give feedback ^^. The author (as mentioned in the comments, not the original author), who graciously gave his sources to be worked from, has a handy blog in which describes the file structures and algorithms used on them in detail. That and my mentors' help has been the source of progress.
On a personal note however, regarding development speed, I've spent too much time debugging errors in 30 lines of code today and boy was it frustrating. In my personal opinion I'm making progress very slowly, I would rather it went a lot faster, I'll have to work on being more efficient with my time in the future or I'll finish nothing be the end of the summer.
Regarding other stuff, git has repeatedly got out of my control and me being me, I've gone the easy way and used force push. I'm really glad nobody is working on the same branch, because otherwise I'd have been burned at the stake by now, but hey, at least my history is clean. :D
Edit: As per request, some image to go with the post, (the palette is miss matched, but it's still something).
Although I have never officially announced my work for classic point and click adventure game The Gene Machine, I have already finished the soundtrack… Just one problem, before I release it in full, I need permission. I have always sought permission to release my enhanced soundtracks, however tracking down those who have the rights to allow me to do just that has proven… well… problematic!
To whet your whistle however, I have decided to release a single track as a preview in the hope that one day, the powers that be will contact either myself or the ScummVM team allowing this to happen.
Until then, enjoy this little taster as we all keep our fingers crossed.
Since the 1.8.0 release we've been very busy with fixing small and big bugs in the newly supported engines,
and now we are almost ready to present you the result in a form of the ScummVM 1.8.1.
Please, help us to test a few games, especially those which were not touched for a long time,
or those which have significant changes in their engine. We prepared a short list on
Also from now on we encourage you to test any game which we support and report any bugs or
your success on the forum so we can keep track of current state of affairs in the ScummVM Land.
For testing the pre-release you need to download a daily build
of ScummVM. If you spot some glitches or bugs, please report them on our
bug tracker. If everything went smoothly,
please report your success on the forums,
and we will instantly reflect it on the wiki
(yes, you can be famous for your testing efforts!) You can find a guide on how to test a new release
A few games are still missing screenshots. Please
help up complete our gallery.
One of the most important tasks for my project is to design the API right. That’s something I would start with, so if there are any mistakes or wrong decisions, I’d be suffering from those later. Of course, I don’t want to suffer or rewrite the API over and over again, so I should think it through.
I already mentioned I wrote a small prototype, which uses Dropbox API and has simple sync feature. I actually thought I should write prototypes for all cloud providers I’m going to add support of, so I get experience of using those. That would help me find common parts and differences, and with all this experience I would be ready to design a good API — considering all pitfalls I met writing prototypes.
But a few days back I thought that I can just imagine an «ideal» API: something simple, designed exactly for my goal. What do I need? I need listing files, uploading and downloading. OK, I also would need deleting, updating modification date and creating directories. And... that’s it. And, as that’s an API, I don’t care how it’s implemented. I care that backend does these operations, no matter what it does in order to do these. (Even though it’s me who will be writing these backends.)
I still would study Google Drive and OneDrive APIs, and may be I’ll get some free time in order to write prototypes for those too. But for now I’ve designed two simple sketches I’ve shown to my mentor, Peter Bozsó (uruk-hai), and Eugene Sandulenko (sev).
First one is quite similar to what I’m using in my prototype:
bool upload(path, file_contents) //may be "save"
file_contents download(path) //may be "read"
bool touch(path) //update modification date
service_info info() //username, available disk space, etc
In this one file is a struct with a few important fields. What I need is path, size, timestamp and a flag to know whether this file is a directory. No service-specific file id and stuff. All the operations are declared in service and different backends implement those, using this common file struct.
The other is a little bit more object-oriented:
bool touch() //update modification date
service_info info() //username, available disk space, etc
There file has a reference to the service it belongs to and includes all file-related operations. That means I would have to implement file backends as well.
The thing is I’m fine with both of these approaches, and mentors opinions were divided. So, they advised me to write this post and get some feedback about it.
By the way, I also have an idea of making a backend for local filesystem as well. There is an AbstractFS in ScummVM, but I think it would be better to use one small Wrapper over it in order to work with local files in terms of the same API I’m working with remote ones.
On Friday Google announced the accepted projects for this year's Summer of Code. We are pleased to say that ScummVM will be mentoring four lucky students who will have to spend their summer in a dark room in front of their screen. We expect this will be an interesting Journey, passing through shadowy gates to visit scary dungeons under dark clouds. All while the mentors will be hunting three-headed monkeys on a tropical island, enjoying the sun, the beach and the grog.
Borja Lorente Escobar (a.k.a. blorente) will be working on porting the MacVenture engine to ScummVM.
Bendegúz Nagy (a.k.a. WinterGrascph) will work on adding support for Dungeon Master
Dmitriy Iskrich (a.k.a. Iskrich) will be working on a Macromedia Director engine, with focus on The Journeyman Project
Ткачёв Александр (a.k.a. Tkachov) will be adding cloud storage integration to ScummVM to allow sharing saves and game data between devices.
Welcome to our students for what we hope will be a productive and interesting summer! You can follow their progress throughout the summer on the ScummVM Blogs.
I've have been more than delighted that I have been accepted to GSoC! My excitement knew no bounds when I got the email.
This is my very first post and my very first blog actually. Cranking out even short texts out of my fingers has always been a challenge for me, but I'll try nonetheless, hopefully on a weekly basis.
My task will be to build the engine for Dungeon Master so that players can... well play with it again. I'm really glad that ScummVM now includes rpgs, as otherwise I could now have found a project with ScummVM to my liking. According to my schedule I'm am to start working on the project this week already, we'll see how that unfolds. Hopefully my studies won't interfere much and my next blog post here will be of the progress I've made with the engine.
As you may already be aware, this year the ScummVM project is participating to the Google Summer of Code. One of the rules for students who want to participate with us is that they need to submit a simple patch against the ScummVM source code before they are accepted. Usually we direct prospective students to our bug tracker for ideas on what they could implement. But now most of the bugs that are still open are not trivial to fix. So I was looking at the source code hunting for simple things to do when I found a TODO comment I left two years ago when implementing the TaskbarManager API on OS X.
What is the TaskbarManager API?
The TaskbarManager API allows interacting with the ScummVM application icon in the taskbar (or in the case of OS X in the dock). We have implementations of this API for several systems, but the only one that is complete is the implementation for Windows. In details the API allows to:
Display an overlay icon when playing a game. If you have in your extra path png files named after the game IDs, when starting a game the corresponding png image is overlaid on the ScummVM icon in the dock. You can for example get icons from http://www.megamonkey.org/icons/ and below is example of this feature in action.
Display progress. This is for example used in the mass add feature to indicate the number of directories scanned in respect with the total number of directories to scan.
Display a count. This is also used in the mass add feature to indicate how many games are being added.
Notify of an error.
Provide a list of recently played games.
The last two were not implemented on OS X and the TODO comment was related to the last point. The idea was that we could provide a list of recently played games in the ScummVM dock menu and thus provide a shortcut to start a game quickly. I decided to take another look at this feature and wrote some bits of code to check that the idea I had hinted at in the TODO would indeed work. I then waited a few days in case a student wanted to implement this as part of his application to the GSoC. But today I finished implementing this feature, cleaned the code and pushed this to the ScummVM repository.
One aspect to consider here is that we want to customise the menu on the ScummVM icon in the dock when ScummVM is not running. That way we can propose a list of recent items in the menu and start ScummVM directly with a game. On OS X we can provide this feature with a plug-in that implements the NSDockTilePlugIn protocol. If an application bundle contains such a plug-in, the OS loads that plug-in when the application is added to the Dock. So there are actually two separate things to implement:
Obviously we need to implement the plug-in.
But we also need to implement code in ScummVM to update the list of recent games when starting a game.
Saving the list of recent games
The TaskbarManager is part of the ScummVM application and when starting a new game the addRecent method is called. So what I did here was simply to save the list or recent games in a place where the aforementioned plug-in can find it. I decided to use the NSUserDefaults class to do this, which means the list is saved in the user preferences (to be precise in the ~/Library/Preferences/org.scummvm.scummvm.plist file).
(if you don't see the source code below visit the blog as it may not be visible in RSS feeds)
That code is a bit too simple though. There are two main issues with it: the list can grow indefinitely and the same game can appear multiple times in the list. So let's improve the code that updates the array of games.
And that's it. We have this part fully implemented. After playing a few games the ~/Library/Preferences/org.scummvm.scummvm.plist file should look like this:
Implementing the NSDockTilePlugIn
If you took a look at the NSDockTilePlugIn protocol documentation you will have seen that it requires implementing a setDockTile: method, and optionally we can implement a dockMenu method. We actually have nothing to do in the first one, so let's skip it and look directly at the second method.
Here we can note that I am using CFPreferences to read the list of recent games and not NSUserDefaults. Why is that? Do I need to remind you that this code is in a plug-in and not in ScummVM? That means we need to access the preferences of another application. Admittedly we could have used NSUsersDefault addSuiteNamed: to achieve this, but remember, we are implementing a plug-in and not an application. The plug-in is loaded by the SystemUIServer and using NSUserDefaults addSuiteNamed: would have changed the global preferences domain list for the SystemUIServer and not only for the plug-in.
The second point we can note is that the code above is using something called StartGameMenuItem. As you have probably guessed this is a custom class that derives from NSMenuItem. Indeed for each menu item I needed to store somewhere the game ID so that when this menu item is activated it can start the corresponding game. So I decided to inherit from the NSMenuItem class and store the game ID in the derived class. And while I was at it I also added the method to start a new game in that derived class. So here is what this class looks like:
Now if you add ScummVM to the dock, and after playing at least one game, you should see the list of games you played recently in the the dock menu like in the picture below. This provides a quick way to start one of those games.
And here is what it looks like in action:
This is in my opinion the most useful of the features provided by the TaskbarManager API, so I am happy to see it finally implemented (I would have done it sooner if I had not forgotten about it :P).
Edit: Our buildbot uses an older SDK that does not support the NSDockTilePlugIn protocol. So nightly builds from our web site will not contain this new feature. You will need to compile your own version or wait for ScummVM 1.9.0.
Google Summer of Code is an inititative by Google to join Open Source organizations, and CS students, so that new talent can get a taste of a professional working environment, while developing some (hopefully) valuable project for the organizations.
I applied for the program this year, inspired by my friend and C++ savant Manu Sánchez, and I got in :)
ScummVM is an organization dedicated to the preservation of old adventure games. I choose to apply to ScummVM for various reasons:
First, there’s the community. From the moment I walked into the IRC, I found one of the most welcoming and understanding communities I’ve been to. So much so that, even though it soon became obvious that I was far from “knowing how to C++”, they helped me set up a non-supported dev environment through 5 hours of trial and error.
There is not much to say about my project that hasn’t been said elsewhere, so I’ll just say that it entails a good deal of reading other people’s code, and butchering it to do some software design black magic with it :)
So, let’s do this
I believe there is nothing else to say, except that I’m really eager to begin this amazing experience, and that I’ll give it my best.
Game has seemed to me extremely interesting, pseudo 3D mixed up with logical riddles.
In the middle of the way I want to realize work with bookshelf animation. It is file, common in Director, so called “movie”. From files of this kind it is possible to receive sounds, images, text, scripts, information about time/frame rate, arrangement of objects, etc. Thus I can allocate the following tasks for 1st milestone:
Provide loading animation files.
Drawning simple objects, like bitmaps, text, shapes.
Organize work of animation cycle. (tempo, FPS, delay moments)
Execute simple Lingo scripts.
As result of middle point, fuzzie suggest to test simple early game. I completely support this decision, but I am not sure about concrete game.
OK, so that’s the first post. Looks like this separate GSoC blog page works like I want it to.
I’m participating in Google Summer of Code for the second time, and today I’ve learned that my proposal to ScummVM was accepted! I’m going to add cloud storage integration into ScummVM, so users would be able to sync their saves and game data between their devices.
I wasn’t posting anything about my work last year (even though I mentioned I probably would in my proposal), so I don’t have a lot of experience writing blog posts in English (actually, I don’t write blog posts in Russian too often as well). Keeping this blog is one of the requirements of ScummVM, so I’d be posting something there at least every week during GSoC.
I’m a little bit busy these days with all my studies, but I had some free time a few weeks back, so I made the first prototype. It has no GUI and it can use Dropbox to upload or download files. I also wrote some basic syncing, which works with files and directories (meaning if you have old file on local machine and a directory with newer files on remote server and names of this file and that directory are the same, it would correctly remove the file, create a directory and then download its contents).
And I guess that’s it for today. I’m not yet sure when I’ll write again.
What began as a casual lunch-time activity (spread across many, many lunches), my work on Douglas Adam's Starship Titanic is finally starting to see significant results. See below for an example of the current progress from the start of the game: https://www.youtube.com/watch?v=8ypLR4fS6vE
You can't get as far as the ship crashing into your house yet, but at least you can fiddle around with the computer and, more importantly, use the television, and see Douglas Adm's visage scolding you to get on with the game. :)
I was originally attracted to working on the game more for technical reasons then the actual gameplay for several reasons. Firstly, because it was a Windows game.. my previous disassembly work has all been on DOS games, so I thought it would make a nice change of pace. Secondly, the original executable relies on compatibility tweaks to run on modern Windows systems and, according to the GOG forums, multiple people have had trouble getting it to run at all. And thirdly, the ease of disassembly.
The game has, in my opinion, the cleanest and most well thought class structures of any game I've worked on. Part of this clean hierarchy involved the bulk of classes in the game deriving from a common "CSaveableObject" base class that defines, amongst other things, the name of the class. The entire game state is laid out as a tree sturcture, with CRoomItem objects for rooms, CNodeItem objects for positions within a room, CViewItem for the different directional views within a node, and game objects under each view. Saving and loading games, including loading the initial game state for new games, is then a simple matter of dumping the hierarchy to and from disk.
Because of this, it's made it somewhat easier to reverse engineer how the classes are implemented. Since the game loading code needs to be able to locate and create classes by their textual name, it meant that I was able to properly identify and name all the classes the same way they were in the original source code (which I don't have access to). Since the original uses C++ classes and inheritance, it's meant that when I've identified the meaning of a virtual method, I could apply the same name to the same method in all the other objects. So, for exmaple, when I identified the methods for saving and loading an item's fields, I was able to focus on those same methods in all the other classes to handle loading the entire game state.
So as you can see from the above video, progress on the engine is going well. I have all the core logic in place for loading the initial game state, displaying views, and moving between them. I've also been able to graft some of the original's movie class, that handles playing AVI videos for all the game's animations, to use the existing ScummVM AviDecoder. Although there's still parts of CMovie that I don't understand. And there also isn't any background sound yet. Apart from that, there's two main areas left to be handled: firstly, all the hardcoded game logic. If there's one thing that I find a pity, it's that rather than using some kind of scripting system, they chose to implement all the game logic in code. So all the various interactable objects in the game have their own code that will need to be slowly implemented.
The other main area remaining to be figured out is the PET control that provides the game's user interface. Particularly the conversation system, where you can converse with the characters within the game. I've already made some minor in-roads into implementing display of the PET, and various error messages in the original executable have given me some ideas of various classes in the conversation system. But it will still take a while to fully implement the game. Plus of course, dispute recently working on it quite a bit in my off hours, work on this game has primarily been a lunch-time diversion, so it's somewhat limited by my work's totally unreasonable policy that lunch should be limited to only a single hour each day :)
We are pleased to announce that John Carmack (yes, the John Carmack) will be joining the ScummVM and ResidualVM teams.
We first met John at GDC 2016 and hit it off immediately.
We had been feeling for a while now that we have to move beyond simply recreating old games and actively improving them, while staying true to their original spirit.
We talked to John about our plan after a few tequilas and he was immediately on board.
John's experience will be invaluable in helping us achieve our goal.
Here is screenshot of John's early work with Monkey Island - notice the improved combat system and the subtle tweaks to Captain Smirk's sprite, you'll see why we are so excited.
If this sounds like something you would be interested in, please take a look at the
merged list of ideas,
the ResidualVM ideas are at Game Tasks section.
You can also propose projects of your own; we'll be happy to help you shape your idea into a
concrete project. If you want to participate or have questions about GSoC, come talk to us on IRC
Freenode channels #residualvm or #scummvm.
It is hard to imagine a company lasting 25 years, especially one focusing on games, however that is exactly the length of time Revolution Software have been delighting us with their fantastic selection of titles over the last quarter of a century.
To celebrate, in March they are releasing a vast box set with all of their released games and a number of interviews and assets for good measure. This box set also features an addition which has surprised me greatly. They have also included a selection of the enhanced music tracks from Beneath A Steel Sky I created as part of my ScummVM Music Enhancement Project. ‘Flattered’ is certainly the word!
From the official website, they have posted…
To celebrate over a quarter of a century of creating adventure games, we’re proud to announce the forthcoming release of the Revolution: the 25th Anniversary Collection box set on 11th March 2016, exclusively for PC.
As well as featuring each of Revolution’s full roster of games, the box set will also contain comic books, posters, high quality audio tracks, a Broken Sword USB, a digital timeline with extensive behind- the-scenes materials, and two hours of video telling the Revolution story. The video footage features interviews with key team members and our creative partners such as Dave Gibbons, Rolf Saxon and Barrington Pheloung. Revolution official site
The end of the year is approaching fast, and just like last year, I want to use this time for some retrospection.
First of all, what happened in the last year?
berenmadded support for building xoreos with CMake, by the way of parsing the automake files used for the autotools build system. This way, xoreos can now be built with either CMake or autotools. I was skeptical at first, especially since I harbour no love for CMake, but it is working reasonably well and I am quite happy with it. In hindsight, I was wrong to reject this pull request for so long.
I focused on supporting all the different model formats used in the Aurora games, and then I made all the games display their in-game areas with objects.
xoreos adopted the Contributor Covenant as its Code of Conduct, in the hopes that it helps foster a friendly and welcoming community.
The big one: our first official release, xoreos 0.0.2, nicknamed “Aribeth”.
I overhauled the script system, making it more generic. This way, I was able to apply it to all targeted games, except Sonic Chronicles: The Dark Brotherhood (which doesn’t seem to use any scripts at all). This included figuring out and implementing four new script bytecode opcodes: two for array access in Dragon Age: Origins, and two for reference creation in Dragon Age II.
I implemented reflective environment mapping for Neverwinter Nights and the two Knights of the Old Republic games.
I added a new tool to the xoreos-tools package: xml2tlk, which can recreate TLK talk table files out of XML files created by tlk2xml.
With these changes, I decided to push out xoreos 0.0.3, nicknamed “Bastila”.
This is all old news, more or less already discussed in previous blog posts. However, since then, I added yet another new tool to the xoreos-tools package: ncsdis. It’s a disassembler for NCS files, the stack-based compiled bytecode of the C-like NWScript, BioWare’s scripting language used throughout their Aurora-based games.
It basically replaces the disassembler within the old OpenKnightsN WScript compiler, with various added benefits. I’ll write a bit more about this tool in the near future, so for now I’ll just leave you with an example assembly listing it can produce, as well as a control flow graph it can create (with the help of Graphviz). As you can see, it already groups the instruction by blocks and subroutines. It performs a static analysis of the stack (to figure out subroutine parameters and return types) and it also analyzes the control flow to detect assorted control structures (loops, if/else). I plan to grow it into a full-fledged NWScript decompiler.
Additionally, I also added support for BioWare’s Neverwinter Nights premium modules, like Kingmaker, to xoreos.
On the documentation side of things,
I added comments and documentation to various files in the xoreos sources, hopefully making them more understandable and useful for potential new contributors and otherwise interested people. Considering how awful my memory is at, this is also a kind of future-proofing.
Farmboy0 added “research” subpages for various games on our wiki, filling them with information about their workings.
I am thankful to all the people in the different BioWare modding communities, for having figured out many different things already. Skywing for example, who had emailed me a few years ago about certain NWScript issues, issues I recently stumbled over again.
I am thankful to fuzzie, for giving me pointers on the NCS disassembler/decompiler.
I am thankful to the GamingOnLinux people, who do a lot of work reporting on all sorts of Linux-related gaming news, and who so graciously mirror my xoreos blog posts.
I am thankful to kevL, for notifying me of issues with xoreos’ build system on configurations I hadn’t thought about.
I am thankful to clone2727, for putting up with rants and ravings.
Finally, after many years of half-hearted attempts, I've finally rewritten my RTLink decode tool practically from scratch to handle disassembling the RTLink/Plus overlay management used by the later Legend entertainment games. See rtlink_decode for the result. The original version of the tool was pretty dodgy, and pretty much hardcoded to only handle the MADS games (Rex Nebular, Return of the Phantom, and Dragonsphere). In this posting, I'll go into more detail of what RTLink/Plus was for those who may be interested.
In the latter days of DOS gaming, game developers started running into a problem. Namely, that their executables were starting to hit the limits of main memory. Not every game could, or would, take advantage of scripting game content, so having all the game logic in the main executable caused the size to bloat. So what to do if your executable was now too big to fit in memory? This was the problem RTLink/Plus was designed to solve.
RTLink is essentially an overlay manager. It splits a program's compiled code into multiple different segments, and allows them to be loaded in as needed, and then replaced when code in other segments needs to execute. RTLink can handle recursive segments, with individual segments split up into their own set of swappable sub-segments. It also allows for multiple different "loading areas" in memory that can independently have their own set of segments. See this article for more general information about RTLink.
Before I go into more details of the problem this posed for disassembly, let's go over how RTLink implements the overlay manager in code. So far, I've encountered three different variations on RTLink being used in executables. What I'll call variation 1 & 2 seem to be the most common form of RTLink in games I've examined. When a program is compiled with either of these versions of RTLink/Plus, one of the segments in the code will contain the RTLink logic, as well as two main areas: the dynamic segments and the function thunks.
The segment list is a list of the dynamic segments within the application. It contains the following information:
The segment in memory where the dynamic segment should be loaded
Whether the segment is stored in the application or a secondary overlay file (Variation 1 only, version 2 only ever uses the executable).
The file offset and size of the segment
The number of relocation entries the segment has; Variation 1 only. Variation 2 has it as part of the starting header for the segment pointed to.
When a segment is needed, the above details are used to read the segment's data from file, and load it into the correct place in memory. The data for a segment consists of two parts: an initial header area, and the date/code for the segment. For variation 1, the header area simply consists of a list of relocation entries. Whereas for variation 2, the details of segment size and number of relocations are provided in a header at the start of the segment, before the relocations list.
A segment's relocation entries are used for the same purpose as relocation entries in a standard application - executables can be loaded at different locations within memory, so all segment references need to be relative to the starting point of where the program is loaded. By keeping the relocation entries for each dynamic segment together with the segment data itself, it's easier for RTLink to apply any needed segment adjustments each time a dynamic segment is loaded.
This is fine to handle shifting the segments in and out of memory, and allow them to have valid memory references, but what causes them to be loaded? The answer is the method "thunks" area of the RTLink segment. When dealing with dynamic segments, you can't just do a far call to some offset in the area of memory segments are loaded in.. you couldn't be sure that the segment you want is actually loaded, or still in memory and not unloaded by some other segment. For this purpose, the thunk list is present.
For every method in a dynamic segment that is referenced by any other segment, a thunk/stub method is created. These consist essentially of the following: a call to the RTLink manager to load the correct segment for the method, a far jump to the method in the correct memory location in the loaded segment, and a following 16-bit value specifying which segment the thunk is for. This way, the thunk method acts as a wrapper, ensuring the correct segment is loaded and passing control to the method to execute.
For variations 1 and 2, the thunk methods have some minor differences, such as version 2 using far calls to the RTLink segment loading code, and having an optional word after the segment index. The segment selector in the far jump call is also already loaded with the memory segment in variation 1, whereas in version 2 it's normally 0 initially, and then set to the correct segment when the thunk method is called. This allows variation 2 to dynamically load the segment in different places in memory, whereas variation 1 is limited to a single specific loading point.
The RTLink segment loader method also mucks around with the stack to push a new intermediate return address on the stack for when the method that's jumped to finishes. This return address points to a code fragment that also handles the case where a method in a dynamic segment calls a method in another one.. in that case, it handles reloading the original segment, so that the original caller's code can be safely returned to.
Put altogether, this scheme allows programs of practically any size needed. As the program grows, the code simply needs to be split into more and more dynamic segments which will get loaded only when needed, and remain on disk when not. Great for having big programs, but not so great for those of us interested in reverse engineering the game by disassembling the executable.
There were several problems to be solved for disassembling such games, which I'll go into now.
A standard IDA disassembly doesn't have all the code
Well, it wouldn't. If you try to disassemble an RTLink/Plus compiled game, IDA will give you an error about unused data at the end of the executable. This will be for one or more RTLink segments. Additionally, as previously mentioned, some of the code for the program can also be stored in a separate OVL (Overlay) file.
Well, I could just load the raw data for them into the disassembly, right?
Well, no. That wouldn't help much, because of all the thunk methods. They all have their references to the same area of memory where segments are expected to be loaded. If you were doing things manually, you'd need to get the details of each segment from RTLink, manually load the code and/or data into new IDA segments, and then manually adjust the thunk methods to point to those methods.
You'd also need to worry about the dynamic segment relocation entries. If you manually loaded the code for a segment, you'd have to read the list of relocation entries for the dynamic segment and manually adjust each relocation entry within the segment. Segment selectors may point to code within the segment (or another sub-segment within the loaded overall RTLink segment), to a low memory area of the executable that remains static in memory, or to the data segment (at a higher memory segment). All in all, you'd have to be extraordinarily patient to all that by hand.
So that's why you wrote rtlink_decode, right? That's what it does?
Yes and no. A bit part of what it does is indeed doing the above to create a new executable suitable for disassembly. This includes laying out all the dynamic segments sequentially (without their segment headers and relocation lists), handling relocation fixups, and the thunk methods adjusted to point to their methods in the decoded executable. However, another problem crops up in the handling of the data segment.
In my experience with RTLink, I've come across across two types of data segments:
In the case of the later Legend Entertainment games, the executable has a single RTLink segment, with the remainder of the segments coming from an OVL file. The single executable segment is for the main data segment as well as a few other miscellaneous segments.
In the case of the MADS games, the data segment isn't an RTLink segment, but all the RTLink segments follow it in the executable.
In both cases, we have a problem doing a proper disassembly. Executables are normally expected to have the data segment at the end of the program, because the data segment may be longer than the end of the executable. For example, a game's data segment may only have 1Kb of pre-set values which are stored in the executable, but it still requires 40Kb of unallocated/uninitialized space. That's why you'll frequently see, when you do a disassembly of a program, areas at the end of a data segment with '?' mark values, indicating the memory isn't part of the executable, so doesn't have any specific value when the program starts.
So if we did just lay the segments end to end, the data segment, coming before other dynamic segments, would end up being shorter than it should be, and a lot of the references to data within it would end up wrapping onto the following dynamic segments in the reworked executable. To avoid this, the rtlink_decode tool ensures that the data segment falls at the end of the generated executable, after all the other segments. This, however, causes it's own share of problems. All the existing references to the data segment refer to where the data segment was expected to be loaded in memory, not to where the data segment actually is in the new executable. Because of this, all the references to the data segment in the executable have to be adjusted accordingly.
Ouch! Sounds fiddly.
It is. And took a lot of messing around to get right. Even then, that's not the entirety of the picture. For Companions of Xanth, the Legend game I used for testing when rebuilding the tool, the data segment has some extra gotcha's.. It contains segment references into the middle of the memory area RTLink segments are loaded into. Presumably these are used in some special controlled circumstances when a specific segment (or segments) are loaded to access particular data. But it's impossible to know without understanding the game a lot better.
Worse, the presence of the references were screwing up some of the loaded dynamic segments in the disassembly, causing them to be split in half. To handle this, the tool explicitly looks for such "bad" references in the data segment, and removes the relocation entries for them. This way, the value in the data segment will remain as a static word, and the segments don't get incorrectly split up. The user can always then later manually set up a pointer to an appropriate segment if they wish. This handles the bulk of such errors, but Xanth at least, there are still references in the low part of the executable (that remains static in memory) to locations within the RTLink segments. Since I can't know which particular RTLink segment is meant to be loaded when the code they're in is called, these few remaining references will have to be later manually adjusted as well.
So that's it?
Yep. After all these years, I'm finally able to generate a (mostly valid) "decoded" executable, and produce a clean disassembly of Companions of Xanth. I also, initially, had two separate versions of the the tool, one the old hacky version for MADS games, and the legend variation for Legend-style RTLink usage. I've since updated my tool to properly handle MADS games, so now there's only the single rtlink_decode tool, and it can handle both variations 1 and 2.
Oh, wait.. what about the 3rd variation you mentioned?
Ah, yes, I didn't really get into that, did I. This version seems to be somewhat different than the other two variations. In this case, the RTLink code is stored in a separate rtlinkst.com file, and then loaded into memory. It then shifts part of the program downwards in memory, and uses it's own relocation table to manually process relocation entries on the shifted code. This variation is proving tricky to disassemble, so whilst I have located the segment list, I still need to:
Figure out how relocation data is encoded. I think I've located the correct data in the executable, but the code RTLink uses to update relocation entries is pretty nasty and overcomplicated.
How much of the start of the executable to remove so that the produced executable doesn't have any of the old code at the start of the executable that gets overwritten
Find the thunk methods, and see whether the existing code will handle them.
Hopefully I can quickly figure out the remaining details for the third variation soon. The goal is to have a tool that both myself and others can use in the future to help them disassemble any game that used RTLink/Plus. Then no-one else will have to go through all the frustrations that I did trying to deal with this %#@! thing.