ScummVM logo ScummVM website - Forums - BuildBot - Doxygen
Contact us - Buy Supported Games: GOG.com
curved edge

PlanetPlanet

Welcome to the ScummVM planet - This aggregates the personal blogs of developers, teams members and active participants from all around the ScummVM community.
If you wish to subscribe to updates to the planet or individual blogs please use the links on the right hand side.
To add your blog to the planet contact DJWillis.

May 02, 2016

ScummVM News Headlines

It is Time to Test ScummVM 1.8.1 Release

Since the 1.8.0 release we've been very busy with fixing small and big bugs in the newly supported engines, and now we are almost ready to present you the result in a form of the ScummVM 1.8.1.

Please, help us to test a few games, especially those which were not touched for a long time, or those which have significant changes in their engine. We prepared a short list on our wiki.

Also from now on we encourage you to test any game which we support and report any bugs or your success on the forum so we can keep track of current state of affairs in the ScummVM Land.

For testing the pre-release you need to download a daily build of ScummVM. If you spot some glitches or bugs, please report them on our bug tracker. If everything went smoothly, please report your success on the forums, and we will instantly reflect it on the wiki (yes, you can be famous for your testing efforts!) You can find a guide on how to test a new release here.

A few games are still missing screenshots. Please help up complete our gallery.

by sev (nospam@scummvm.org) at May 02, 2016 06:00 AM

April 26, 2016

Alexander Tkachev (Tkachov) - GSoC

GSoC: Designing API

One of the most important tasks for my project is to design the API right. That’s something I would start with, so if there are any mistakes or wrong decisions, I’d be suffering from those later. Of course, I don’t want to suffer or rewrite the API over and over again, so I should think it through.

I already mentioned I wrote a small prototype, which uses Dropbox API and has simple sync feature. I actually thought I should write prototypes for all cloud providers I’m going to add support of, so I get experience of using those. That would help me find common parts and differences, and with all this experience I would be ready to design a good API — considering all pitfalls I met writing prototypes.

But a few days back I thought that I can just imagine an «ideal» API: something simple, designed exactly for my goal. What do I need? I need listing files, uploading and downloading. OK, I also would need deleting, updating modification date and creating directories. And... that’s it. And, as that’s an API, I don’t care how it’s implemented. I care that backend does these operations, no matter what it does in order to do these. (Even though it’s me who will be writing these backends.)

I still would study Google Drive and OneDrive APIs, and may be I’ll get some free time in order to write prototypes for those too. But for now I’ve designed two simple sketches I’ve shown to my mentor, Peter Bozsó (uruk-hai), and Eugene Sandulenko (sev).

First one is quite similar to what I’m using in my prototype:

file {
    path
    name
    size
    timestamp
    is_directory
}

service {
    file[] list_directory(path)
    bool upload(path, file_contents)    //may be "save"
    file_contents download(path)        //may be "read"
    bool delete(path)
    bool sync(service)
    bool create_directory(path) 
    bool touch(path)                    //update modification date
    service_info info()                 //username, available disk space, etc
    bool is_syncing()
}

In this one file is a struct with a few important fields. What I need is path, size, timestamp and a flag to know whether this file is a directory. No service-specific file id and stuff. All the operations are declared in service and different backends implement those, using this common file struct.

The other is a little bit more object-oriented:

file {
    path
    name
    size
    timestamp
    is_directory
    service

    file[] list()
    file_contents get_contents()
    bool set_contents(file_contents)
    bool delete()
    bool touch()            //update modification date
    bool sync(file)
}

service {
    file get_root_directory()
    file get_saves_directory()
    bool create_directory(path) 
    service_info info()     //username, available disk space, etc
    bool is_syncing()
}

There file has a reference to the service it belongs to and includes all file-related operations. That means I would have to implement file backends as well.

The thing is I’m fine with both of these approaches, and mentors opinions were divided. So, they advised me to write this post and get some feedback about it.

By the way, I also have an idea of making a backend for local filesystem as well. There is an AbstractFS in ScummVM, but I think it would be better to use one small Wrapper over it in order to work with local files in terms of the same API I’m working with remote ones.

by Tkachov at April 26, 2016 08:00 PM

April 25, 2016

Bendegúz Nagy (WinterGrascph) - GSoC

GSoC Acceptance

I've have been more than delighted that I have been accepted to GSoC! My excitement knew no bounds when I got the email.

This is my very first post and my very first blog actually. Cranking out even short texts out of my fingers has always been a challenge for me, but I'll try nonetheless, hopefully on a weekly basis.

My task will be to build the engine for Dungeon Master so that players can... well play with it again. I'm really glad that ScummVM now includes rpgs, as otherwise I could now have found a project with ScummVM to my liking. According to my schedule I'm am to start working on the project this week already, we'll see how that unfolds. Hopefully my studies won't interfere much and my next blog post here will be of the progress I've made with the engine.

That concludes my very first blog post.

by Bendegúz Nagy (noreply@blogger.com) at April 25, 2016 07:44 AM

ScummVM News Headlines

The students are known for GSoC 2016!

GSoC 2016 Banner

On Friday Google announced the accepted projects for this year's Summer of Code. We are pleased to say that ScummVM will be mentoring four lucky students who will have to spend their summer in a dark room in front of their screen. We expect this will be an interesting Journey, passing through shadowy gates to visit scary dungeons under dark clouds. All while the mentors will be hunting three-headed monkeys on a tropical island, enjoying the sun, the beach and the grog.

  • Borja Lorente Escobar (a.k.a. blorente) will be working on porting the MacVenture engine to ScummVM.
  • Bendegúz Nagy (a.k.a. WinterGrascph) will work on adding support for Dungeon Master
  • Dmitriy Iskrich (a.k.a. Iskrich) will be working on a Macromedia Director engine, with focus on The Journeyman Project
  • Ткачёв Александр (a.k.a. Tkachov) will be adding cloud storage integration to ScummVM to allow sharing saves and game data between devices.

Welcome to our students for what we hope will be a productive and interesting summer! You can follow their progress throughout the summer on the ScummVM Blogs.

by criezy (nospam@scummvm.org) at April 25, 2016 06:00 AM

Thierry Crozat (criezy)

NSDockTilePlugIn for ScummVM on OS X

As you may already be aware, this year the ScummVM project is participating to the Google Summer of Code. One of the rules for students who want to participate with us is that they need to submit a simple patch against the ScummVM source code before they are accepted. Usually we direct prospective students  to our bug tracker for ideas on what they could implement. But now most of the bugs that are still open are not trivial to fix. So I was looking at the source code hunting for simple things to do when I found a TODO comment I left two years ago when implementing the TaskbarManager API on OS X.

What is the TaskbarManager API?

The TaskbarManager API allows interacting with the ScummVM application icon in the taskbar (or in the case of OS X in the dock). We have implementations of this API for several systems, but the only one that is complete is the implementation for Windows. In details the API allows to:

  • Display an overlay icon when playing a game. If you have in your extra path png files named after the game IDs, when starting a game the corresponding png image is overlaid on the ScummVM icon in the dock. You can for example get icons from http://www.megamonkey.org/icons/ and below is example of this feature in action.
  • Display progress. This is for example used in the mass add feature to indicate the number of directories scanned in respect with the total number of directories to scan.
  • Display a count. This is also used in the mass add feature to indicate how many games are being added.
  • Notify of an error.
  • Provide a list of recently played games.

The last two were not implemented on OS X and the TODO comment was related to the last point. The idea was that we could provide a list of recently played games in the ScummVM dock menu and thus provide a shortcut to start a game quickly. I decided to take another look at this feature and wrote some bits of code to check that the idea I had hinted at in the TODO would indeed work. I then waited a few days in case a student wanted to implement this as part of his application to the GSoC. But today I finished implementing this feature, cleaned the code and pushed this to the ScummVM repository.

One aspect to consider here is that we want to customise the menu on the ScummVM icon in the dock when ScummVM is not running. That way we can propose a list of recent items in the menu and start ScummVM directly with a game. On OS X we can provide this feature with a plug-in that implements the NSDockTilePlugIn protocol. If an application bundle contains such a plug-in, the OS loads that plug-in when the application is added to the Dock. So there are actually two separate things to implement:

  • Obviously we need to implement the plug-in.
  • But we also need to implement code in ScummVM to update the list of recent games when starting a game.

Saving the list of recent games

The TaskbarManager is part of the ScummVM application and when starting a new game the addRecent method is called. So what I did here was simply to save the list or recent games in a place where the aforementioned plug-in can find it. I decided to use the NSUserDefaults class to do this, which means the list is saved in the user preferences (to be precise in the ~/Library/Preferences/org.scummvm.scummvm.plist file).

(if you don't see the source code below visit the blog as it may not be visible in RSS feeds)


That code is a bit too simple though. There are two main issues with it: the list can grow indefinitely and the same game can appear multiple times in the list. So let's improve the code that updates the array of games.


And that's it. We have this part fully implemented. After playing a few games the ~/Library/Preferences/org.scummvm.scummvm.plist file should look like this:



Implementing the NSDockTilePlugIn

If you took a look at the NSDockTilePlugIn protocol documentation you will have seen that it requires implementing a setDockTile: method, and optionally we can implement a dockMenu method. We actually have nothing to do in the first one, so let's skip it and look directly at the second method.


Here we can note that I am using CFPreferences to read the list of recent games and not NSUserDefaults. Why is that? Do I need to remind you that this code is in a plug-in and not in ScummVM? That means we need to access the preferences of another application. Admittedly we could have used NSUsersDefault addSuiteNamed: to achieve this, but remember, we are implementing a plug-in and not an application. The plug-in is loaded by the SystemUIServer and using NSUserDefaults addSuiteNamed: would have changed the global preferences domain list for the SystemUIServer and not only for the plug-in.

The second point we can note is that the code above is using something called StartGameMenuItem. As you have probably guessed this is a custom class that derives from NSMenuItem. Indeed for each menu item I needed to store somewhere the game ID so that when this menu item is activated it can start the corresponding game. So I decided to inherit from the NSMenuItem class and store the game ID in the derived class. And while I was at it I also added the method to start a new game in that derived class. So here is what this class looks like:


To Conclude

Now if you add ScummVM to the dock, and after playing at least one game, you should see the list of games you played recently in the the dock menu like in the picture below. This provides a quick way to start one of those games.



And here is what it looks like in action:


This is in my opinion the most useful of the features provided by the TaskbarManager API, so I am happy to see it finally implemented (I would have done it sooner if I had not forgotten about it :P).

Edit: Our buildbot uses an older SDK that does not support the NSDockTilePlugIn protocol. So nightly builds from our web site will not contain this new feature. You will need to compile your own version or wait for ScummVM 1.9.0.

by Thierry Crozat (noreply@blogger.com) at April 25, 2016 03:16 AM

April 24, 2016

Borja Lorente (blorente) - GSoC

So... I got into GSOC 2016

… and I still cannot believe it.

Google Summer of Code is an inititative by Google to join Open Source organizations, and CS students, so that new talent can get a taste of a professional working environment, while developing some (hopefully) valuable project for the organizations.

I applied for the program this year, inspired by my friend and C++ savant Manu Sánchez, and I got in :)

cat

ScummVM

ScummVM is an organization dedicated to the preservation of old adventure games. I choose to apply to ScummVM for various reasons:

First, there’s the community. From the moment I walked into the IRC, I found one of the most welcoming and understanding communities I’ve been to. So much so that, even though it soon became obvious that I was far from “knowing how to C++”, they helped me set up a non-supported dev environment through 5 hours of trial and error.

Second, the technical challenges. ScummVM has to support several… um… unusual devices at once (NDS and PS2…), and thus the org has it’s own implementation of basically everything STL (List, String…), which is obviously very, very interesting to look at. This, coupled with the challenge of understanding ancient game code and some high level JavaScript, is a very stimulating challenge. All in all, it’s an amazing opportunity to “learn how to C++”.

My Project

There is not much to say about my project that hasn’t been said elsewhere, so I’ll just say that it entails a good deal of reading other people’s code, and butchering it to do some software design black magic with it :)

So, let’s do this

I believe there is nothing else to say, except that I’m really eager to begin this amazing experience, and that I’ll give it my best.

April 24, 2016 01:25 PM

April 23, 2016

Dmitriy Iskrich (iskrich) - GSoC

Let’s start our journey with Director!

It is the first post from series of posts about my future summer project: integration Director games.

To be quite frank, Director doesn’t need introduction. It is the game engine popular in due time, from the MacroMind company (Macromedia, Adobe).  You can read in more detail here:

http://lingoworkshop.com/articles/history

On the basis of this platform the huge set of games was developed. I have chosen as the final purpose full integration The Journeyman Project.

https://en.wikipedia.org/wiki/The_Journeyman_Project

Game has seemed to me extremely interesting, pseudo 3D mixed up with logical riddles.

In the middle of the way I want to realize work with bookshelf animation. It is file, common in Director, so called “movie”. From files of this kind it is possible to receive sounds, images, text, scripts, information about time/frame rate, arrangement of objects, etc. Thus I can allocate the following tasks for 1st milestone:

  • Provide loading animation files.
  • Drawning simple objects, like bitmaps, text, shapes.
  • Organize work of animation cycle. (tempo, FPS, delay moments)
  • Execute simple Lingo scripts.

As result of middle point, fuzzie  suggest to test simple early game. I completely support this decision, but I am not sure about concrete game.

At the moment is available:

clone2727, Deledrius work

https://github.com/iskrich/director

In fact loading and parts of parsing is already here, I still should study this code.

And animation player/viewer by fuzzie:

https://github.com/scummvm-director/continuity

There are assumptions that at the moment in C ++ a code not all algorithms of parsing which are in Python code are realized. Therefore one of my first real tasks is the code transfer (Python-> C ++)

At the beginning of May I will study highlights of architecture of the engine. I will begin from here:

http://wiki.scummvm.org/index.php/HOWTO-Engines

In the middle of May I plan to begin studying of the scripting Lingo language and a detail of Director works.

I hope all above-mentioned sounded as the plan.


by iskrich at April 23, 2016 07:52 PM

April 22, 2016

Alexander Tkachev (Tkachov) - GSoC

GSoC: First post

OK, so that’s the first post. Looks like this separate GSoC blog page works like I want it to.

I’m participating in Google Summer of Code for the second time, and today I’ve learned that my proposal to ScummVM was accepted! I’m going to add cloud storage integration into ScummVM, so users would be able to sync their saves and game data between their devices.

I wasn’t posting anything about my work last year (even though I mentioned I probably would in my proposal), so I don’t have a lot of experience writing blog posts in English (actually, I don’t write blog posts in Russian too often as well). Keeping this blog is one of the requirements of ScummVM, so I’d be posting something there at least every week during GSoC.

I’m a little bit busy these days with all my studies, but I had some free time a few weeks back, so I made the first prototype. It has no GUI and it can use Dropbox to upload or download files. I also wrote some basic syncing, which works with files and directories (meaning if you have old file on local machine and a directory with newer files on remote server and names of this file and that directory are the same, it would correctly remove the file, create a directory and then download its contents).

And I guess that’s it for today. I’m not yet sure when I’ll write again.

by Tkachov at April 22, 2016 08:00 PM

April 12, 2016

Paul Gilbert (Dreammaster)

Implementing Starship Titanic in ScummVM

What began as a casual lunch-time activity (spread across many, many lunches), my work on Douglas Adam's Starship Titanic is finally starting to see significant results. See below for an example of the current progress from the start of the game: https://www.youtube.com/watch?v=8ypLR4fS6vE



You can't get as far as the ship crashing into your house yet, but at least you can fiddle around with the computer and, more importantly, use the television, and see Douglas Adm's visage scolding you to get on with the game. :)

I was originally attracted to working on the game more for technical reasons then the actual gameplay for several reasons. Firstly, because it was a Windows game.. my previous disassembly work has all been on DOS games, so I thought it would make a nice change of pace. Secondly, the original executable relies on compatibility tweaks to run on modern Windows systems and, according to the GOG forums, multiple people have had trouble getting it to run at all. And thirdly, the ease of disassembly.

The game has, in my opinion, the cleanest and most well thought class structures of any game I've worked on. Part of this clean hierarchy involved the bulk of classes in the game deriving from a common "CSaveableObject" base class that defines, amongst other things, the name of the class. The entire game state is laid out as a tree sturcture, with CRoomItem objects for rooms, CNodeItem objects for positions within a room, CViewItem for the different directional views within a node, and game objects under each view. Saving and loading games, including loading the initial game state for new games, is then a simple matter of dumping the hierarchy to and from disk.

Because of this, it's made it somewhat easier to reverse engineer how the classes are implemented. Since the game loading code needs to be able to locate and create classes by their textual name, it meant that I was able to properly identify and name all the classes the same way they were in the original source code (which I don't have access to). Since the original uses C++ classes and inheritance, it's meant that when I've identified the meaning of a virtual method, I could apply the same name to the same method in all the other objects. So, for exmaple, when I identified the methods for saving and loading an item's fields, I was able to focus on those same methods in all the other classes to handle loading the entire game state.

So as you can see from the above video, progress on the engine is going well. I have all the core logic in place for loading the initial game state, displaying views, and moving between them. I've also been able to graft some of the original's movie class, that handles playing AVI videos for all the game's animations, to use the existing ScummVM AviDecoder. Although there's still parts of CMovie that I don't understand. And there also isn't any background sound yet. Apart from that, there's two main areas left to be handled: firstly, all the hardcoded game logic. If there's one thing that I find a pity, it's that rather than using some kind of scripting system, they chose to implement all the game logic in code. So all the various interactable objects in the game have their own code that will need to be slowly implemented.

The other main area remaining to be figured out is the PET control that provides the game's user interface. Particularly the conversation system, where you can converse with the characters within the game. I've already made some minor in-roads into implementing display of the PET, and various error messages in the original executable have given me some ideas of various classes in the conversation system. But it will still take a while to fully implement the game. Plus of course, dispute recently working on it quite a bit in my off hours, work on this game has primarily been a lunch-time diversion, so it's somewhat limited by my work's totally unreasonable policy that lunch should be limited to only a single hour each day :)

by Dreammaster (noreply@blogger.com) at April 12, 2016 02:37 AM

April 01, 2016

ScummVM News Headlines

The ScummVM team welcomes John Carmack

We are pleased to announce that John Carmack (yes, the John Carmack) will be joining the ScummVM and ResidualVM teams.

We first met John at GDC 2016 and hit it off immediately.

We had been feeling for a while now that we have to move beyond simply recreating old games and actively improving them, while staying true to their original spirit. We talked to John about our plan after a few tequilas and he was immediately on board.

John's experience will be invaluable in helping us achieve our goal.

Here is screenshot of John's early work with Monkey Island - notice the improved combat system and the subtle tweaks to Captain Smirk's sprite, you'll see why we are so excited.

So please join us in welcoming John!

UPDATE: No, not really. This was a joke.

by The ScummVM Team (nospam@scummvm.org) at April 01, 2016 06:00 AM

March 27, 2016

ScummVM News Headlines

Explore the Myst library

You stumble upon an old handwritten book. It describes an island world named Myst. When you touch one of its last pages, you are transported to the island. As you explore Myst you learn about its inhabitants Atrus, Catherine and their sons Sirrus and Achenar. You quickly discover each son is trapped in a book. Will you solve the puzzles of the Myst ages and free them?

The ScummVM Team is proud to announce that Myst and Myst: Masterpiece Edition are now playable in ScummVM using the latest daily builds, and are ready for testing.

Reports of bugs and unknown versions are appreciated. They should go to our bug tracker, and must follow our bug submission guidelines. You can help us fill our screenshot gallery with captures from your playthrough.

In addition to the main game, the making-of video and the demo version are supported.

by bgK (nospam@scummvm.org) at March 27, 2016 06:00 AM

March 02, 2016

ResidualVM News Headlines

Welcome to Google Summer of Code

GSoC 2016 Logo

We are happy to announce that we are participating in this year's Google Summer of Code under the umbrella of our sister project ScummVM.

If this sounds like something you would be interested in, please take a look at the merged list of ideas, the ResidualVM ideas are at Game Tasks section.

You can also propose projects of your own; we'll be happy to help you shape your idea into a concrete project. If you want to participate or have questions about GSoC, come talk to us on IRC Freenode channels #residualvm or #scummvm.

by aquadran (nospam@residualvm.org) at March 02, 2016 12:00 AM

January 27, 2016

James Woodcock

Revolution’s 25th Anniversary Box Set – My Enhanced Music Featured

Revolution's 25th Anniversary Box-set

It is hard to imagine a company lasting 25 years, especially one focusing on games, however that is exactly the length of time Revolution Software have been delighting us with their fantastic selection of titles over the last quarter of a century.

To celebrate, in March they are releasing a vast box set with all of their released games and a number of interviews and assets for good measure. This box set also features an addition which has surprised me greatly. They have also included a selection of the enhanced music tracks from Beneath A Steel Sky I created as part of my ScummVM Music Enhancement Project. ‘Flattered’ is certainly the word!

From the official website, they have posted…

To celebrate over a quarter of a century of creating adventure games, we’re proud to announce the forthcoming release of the Revolution: the 25th Anniversary Collection box set on 11th March 2016, exclusively for PC.

As well as featuring each of Revolution’s full roster of games, the box set will also contain comic books, posters, high quality audio tracks, a Broken Sword USB, a digital timeline with extensive behind- the-scenes materials, and two hours of video telling the Revolution story. The video footage features interviews with key team members and our creative partners such as Dave Gibbons, Rolf Saxon and Barrington Pheloung.
Revolution official site

//www.youtube.com/watch?v=kVGBbJH_Slc

To find out more, make sure you visit thier website for more details:
> REVOLUTION: THE 25TH ANNIVERSARY COLLECTION BOX SET

Broken Sword 5

by James Woodcock at January 27, 2016 08:52 PM

November 27, 2015

Sven Hesse (DrMcCoy)

xoreos Not-Thanksgiving 2015

xoreos is a FLOSS project aiming to reimplement BioWare’s Aurora engine (and derivatives), covering their games starting with Neverwinter Nights and potentially up to Dragon Age II. This post gives a short update on the current progress.

Note: This is a cross-post of a news item on the xoreos website.

The end of the year is approaching fast, and just like last year, I want to use this time for some retrospection.

First of all, what happened in the last year?

  • berenm added support for building xoreos with CMake, by the way of parsing the automake files used for the autotools build system. This way, xoreos can now be built with either CMake or autotools. I was skeptical at first, especially since I harbour no love for CMake, but it is working reasonably well and I am quite happy with it. In hindsight, I was wrong to reject this pull request for so long.
  • I focused on supporting all the different model formats used in the Aurora games, and then I made all the games display their in-game areas with objects.
  • xoreos adopted the Contributor Covenant as its Code of Conduct, in the hopes that it helps foster a friendly and welcoming community.
  • The big one: our first official release, xoreos 0.0.2, nicknamed “Aribeth”.
  • I overhauled the script system, making it more generic. This way, I was able to apply it to all targeted games, except Sonic Chronicles: The Dark Brotherhood (which doesn’t seem to use any scripts at all). This included figuring out and implementing four new script bytecode opcodes: two for array access in Dragon Age: Origins, and two for reference creation in Dragon Age II.
  • I implemented reflective environment mapping for Neverwinter Nights and the two Knights of the Old Republic games.
  • I added a new tool to the xoreos-tools package: xml2tlk, which can recreate TLK talk table files out of XML files created by tlk2xml.
  • With these changes, I decided to push out xoreos 0.0.3, nicknamed “Bastila”.

This is all old news, more or less already discussed in previous blog posts. However, since then, I added yet another new tool to the xoreos-tools package: ncsdis. It’s a disassembler for NCS files, the stack-based compiled bytecode of the C-like NWScript, BioWare’s scripting language used throughout their Aurora-based games.

It basically replaces the disassembler within the old OpenKnightsN WScript compiler, with various added benefits. I’ll write a bit more about this tool in the near future, so for now I’ll just leave you with an example assembly listing it can produce, as well as a control flow graph it can create (with the help of Graphviz). As you can see, it already groups the instruction by blocks and subroutines. It performs a static analysis of the stack (to figure out subroutine parameters and return types) and it also analyzes the control flow to detect assorted control structures (loops, if/else). I plan to grow it into a full-fledged NWScript decompiler.

Additionally, I also added support for BioWare’s Neverwinter Nights premium modules, like Kingmaker, to xoreos.

On the documentation side of things,

  • I added comments and documentation to various files in the xoreos sources, hopefully making them more understandable and useful for potential new contributors and otherwise interested people. Considering how awful my memory is at, this is also a kind of future-proofing.
  • Farmboy0 added “research” subpages for various games on our wiki, filling them with information about their workings.
  • I extended our TODO list considerably.
  • I added an example configuration file, and extended the documentation on the wiki on how to compile and run xoreos.
  • I wrote man pages for each tool in xoreos and for xoreos itself. I also added the former to the wiki.

Phew! This is again a bigger list than I had anticipated. This wouldn’t have been possible without these people, for whom I am thankful:

  • I am thankful to berenm for providing the CMake bindings, despite my grumbling about it.
  • I am thankful to Supermanu, for continuing on chipping away on the Neverwinter Nights character generator.
  • I am thankful to Farmboy0, for working on xoreos’ Jade Empire engine and researching game internals.
  • I am thankful to mirv, for continuing with the huge task of rewriting my naive OpenGL code.
  • I am thankful to Coraline Ada Ehmke for creating the Contributor Covenant.
  • I am thankful to all the people in the different BioWare modding communities, for having figured out many different things already. Skywing for example, who had emailed me a few years ago about certain NWScript issues, issues I recently stumbled over again.
  • I am thankful to fuzzie, for giving me pointers on the NCS disassembler/decompiler.
  • I am thankful to the GamingOnLinux people, who do a lot of work reporting on all sorts of Linux-related gaming news, and who so graciously mirror my xoreos blog posts.
  • I am thankful to kevL, for notifying me of issues with xoreos’ build system on configurations I hadn’t thought about.
  • I am thankful to clone2727, for putting up with rants and ravings.
  • I am thankful to all the people who told me when I was wrong, for example when I wrongheadedly silenced clang static analyzer warnings, without understanding what I was doing.
  • I am thankful to everybody else who gave me hints and tips, taught me tricks and procedure, showed me new things, old things, forgotten things, broken things.
  • I am thankful to all the people who are not angry with me for forgetting them, because they are aware that this is not meant as a personal slight ;).

Now that I have these mushy feelings out of my system, here’s hoping for another great year! :)

And like always, if you want to join our effort, please don’t hesitate to contact us!

flattr this!

by DrMcCoy at November 27, 2015 02:26 AM

November 04, 2015

Paul Gilbert (Dreammaster)

The intricacies of disassembling RTLink/Plus

Finally, after many years of half-hearted attempts, I've finally rewritten my RTLink decode tool practically from scratch to handle disassembling the RTLink/Plus overlay management used by the later Legend entertainment games. See rtlink_decode for the result. The original version of the tool was pretty dodgy, and pretty much hardcoded to only handle the MADS games (Rex Nebular, Return of the Phantom, and Dragonsphere). In this posting, I'll go into more detail of what RTLink/Plus was for those who may be interested.

In the latter days of DOS gaming, game developers started running into a problem. Namely, that their executables were starting to hit the limits of main memory. Not every game could, or would, take advantage of scripting game content, so having all the game logic in the main executable caused the size to bloat. So what to do if your executable was now too big to fit in memory? This was the problem RTLink/Plus was designed to solve.

RTLink is essentially an overlay manager. It splits a program's compiled code into multiple different segments, and allows them to be loaded in as needed, and then replaced when code in other segments needs to execute. RTLink can handle recursive segments, with individual segments split up into their own set of swappable sub-segments. It also allows for multiple different "loading areas" in memory that can independently have their own set of segments. See this article for more general information about RTLink.

Before I go into more details of the problem this posed for disassembly, let's go over how RTLink implements the overlay manager in code. So far, I've encountered three different variations on RTLink being used in executables. What I'll call variation 1 & 2 seem to be the most common form of RTLink in games I've examined. When a program is compiled with either of these versions of RTLink/Plus, one of the segments in the code will contain the RTLink logic, as well as two main areas: the dynamic segments and the function thunks.

The segment list is a list of the dynamic segments within the application. It contains the following information:
  • The segment in memory where the dynamic segment should be loaded
  • Whether the segment is stored in the application or a secondary overlay file (Variation 1 only, version 2 only ever uses the executable).
  • The file offset and size of the segment
  • The number of relocation entries the segment has; Variation 1 only. Variation 2 has it as part of the starting header for the segment pointed to.
When a segment is needed, the above details are used to read the segment's data from file, and load it into the correct place in memory. The data for a segment consists of two parts: an initial header area, and the date/code for the segment. For variation 1, the header area simply consists of a list of relocation entries. Whereas for variation 2, the details of segment size and number of relocations are provided in a header at the start of the segment, before the relocations list.

A segment's relocation entries are used for the same purpose as relocation entries in a standard application - executables can be loaded at different locations within memory, so all segment references need to be relative to the starting point of where the program is loaded. By keeping the relocation entries for each dynamic segment together with the segment data itself, it's easier for RTLink to apply any needed segment adjustments each time a dynamic segment is loaded.

This is fine to handle shifting the segments in and out of memory, and allow them to have valid memory references, but what causes them to be loaded? The answer is the method "thunks" area of the RTLink segment. When dealing with dynamic segments, you can't just do a far call to some offset in the area of memory segments are loaded in.. you couldn't be sure that the segment you want is actually loaded, or still in memory and not unloaded by some other segment. For this purpose, the thunk list is present.

For every method in a dynamic segment that is referenced by any other segment, a thunk/stub method is created. These consist essentially of the following: a call to the RTLink manager to load the correct segment for the method, a far jump to the method in the correct memory location in the loaded segment, and a following 16-bit value specifying which segment the thunk is for. This way, the thunk method acts as a wrapper, ensuring the correct segment is loaded and passing control to the method to execute.

For variations 1 and 2, the thunk methods have some minor differences, such as version 2 using far calls to the RTLink segment loading code, and having an optional word after the segment index. The segment selector in the far jump call is also already loaded with the memory segment in variation 1, whereas in version 2 it's normally 0 initially, and then set to the correct segment when the thunk method is called. This allows variation 2 to dynamically load the segment in different places in memory, whereas variation 1 is limited to a single specific loading point.

The RTLink segment loader method also mucks around with the stack to push a new intermediate return address on the stack for when the method that's jumped to finishes. This return address points to a code fragment that also handles the case where a method in a dynamic segment calls a method in another one.. in that case, it handles reloading the original segment, so that the original caller's code can be safely returned to.

Put altogether, this scheme allows programs of practically any size needed. As the program grows, the code simply needs to be split into more and more dynamic segments which will get loaded only when needed, and remain on disk when not. Great for having big programs, but not so great for those of us interested in reverse engineering the game by disassembling the executable.

There were several problems to be solved for disassembling such games, which I'll go into now.

A standard IDA disassembly doesn't have all the code

Well, it wouldn't. If you try to disassemble an RTLink/Plus compiled game, IDA will give you an error about unused data at the end of the executable. This will be for one or more RTLink segments. Additionally, as previously mentioned, some of the code for the program can also be stored in a separate OVL (Overlay) file.

Well, I could just load the raw data for them into the disassembly, right?

Well, no. That wouldn't help much, because of all the thunk methods. They all have their references to the same area of memory where segments are expected to be loaded. If you were doing things manually, you'd need to get the details of each segment from RTLink, manually load the code and/or data into new IDA segments, and then manually adjust the thunk methods to point to those methods.

You'd also need to worry about the dynamic segment relocation entries. If you manually loaded the code for a segment, you'd have to read the list of relocation entries for the dynamic segment and manually adjust each relocation entry within the segment. Segment selectors may point to code within the segment (or another sub-segment within the loaded overall RTLink segment), to a low memory area of the executable that remains static in memory, or to the data segment (at a higher memory segment). All in all, you'd have to be extraordinarily patient to all that by hand.

So that's why you wrote rtlink_decode, right? That's what it does?
Yes and no. A bit part of what it does is indeed doing the above to create a new executable suitable for disassembly. This includes laying out all the dynamic segments sequentially (without their segment headers and relocation lists), handling relocation fixups, and the thunk methods adjusted to point to their methods in the decoded executable. However, another problem crops up in the handling of the data segment.

In my experience with RTLink, I've come across across two types of data segments:
  1. In the case of the later Legend Entertainment games, the executable has a single RTLink segment, with the remainder of the segments coming from an OVL file. The single executable segment is for the main data segment as well as a few other miscellaneous segments.
  2. In the case of the MADS games, the data segment isn't an RTLink segment, but all the RTLink segments follow it in the executable.
In both cases, we have a problem doing a proper disassembly. Executables are normally expected to have the data segment at the end of the program, because the data segment may be longer than the end of the executable. For example, a game's data segment may only have 1Kb of pre-set values which are stored in the executable, but it still requires 40Kb of unallocated/uninitialized space. That's why you'll frequently see, when you do a disassembly of a program, areas at the end of a data segment with '?' mark values, indicating the memory isn't part of the executable, so doesn't have any specific value when the program starts.

So if we did just lay the segments end to end, the data segment, coming before other dynamic segments, would end up being shorter than it should be, and a lot of the references to data within it would end up wrapping onto the following dynamic segments in the reworked executable. To avoid this,  the rtlink_decode tool ensures that the data segment falls at the end of the generated executable, after all the other segments. This, however, causes it's own share of problems. All the existing references to the data segment refer to where the data segment was expected to be loaded in memory, not to where the data segment actually is in the new executable. Because of this, all the references to the data segment in the executable have to be adjusted accordingly.

Ouch! Sounds fiddly.

It is. And took a lot of messing around to get right. Even then, that's not the entirety of the picture. For Companions of Xanth, the Legend game I used for testing when rebuilding the tool, the data segment has some extra gotcha's.. It contains segment references into the middle of the memory area RTLink segments are loaded into. Presumably these are used in some special controlled circumstances when a specific segment (or segments) are loaded to access particular data. But it's impossible to know without understanding the game a lot better. 

Worse, the presence of the references were screwing up some of the loaded dynamic segments in the disassembly, causing them to be split in half. To handle this, the tool explicitly looks for such "bad" references in the data segment, and removes the relocation entries for them. This way, the value in the data segment will remain as a static word, and the segments don't get incorrectly split up. The user can always then later manually set up a pointer to an appropriate segment if they wish. This handles the bulk of such errors, but Xanth at least, there are still references in the low part of the executable (that remains static in memory) to locations within the RTLink segments. Since I can't know which particular RTLink segment is meant to be loaded when the code they're in is called, these few remaining references will have to be later manually adjusted as well.

So that's it?

Yep. After all these years, I'm finally able to generate a (mostly valid) "decoded" executable, and produce a clean disassembly of Companions of Xanth. I also, initially, had two separate versions of the the tool, one the old hacky version for MADS games, and the legend variation for Legend-style RTLink usage. I've since updated my tool to properly handle MADS games, so now there's only the single rtlink_decode tool, and it can handle both variations 1 and 2.

Oh, wait.. what about the 3rd variation you mentioned?

Ah, yes, I didn't really get into that, did I. This version seems to be somewhat different than the other two variations. In this case, the RTLink code is stored in a separate rtlinkst.com file, and then loaded into memory. It then shifts part of the program downwards in memory, and uses it's own relocation table to manually process relocation entries on the shifted code. This variation is proving tricky to disassemble, so whilst I have located the segment list, I still need to:
  • Figure out how relocation data is encoded. I think I've located the correct data in the executable, but the code RTLink uses to update relocation entries is pretty nasty and overcomplicated.
  • How much of the start of the executable to remove so that the produced executable doesn't have any of the old code at the start of the executable that gets overwritten
  • Find the thunk methods, and see whether the existing code will handle them.

Hopefully I can quickly figure out the remaining details for the third variation soon. The goal is to have a tool that both myself and others can use in the future to help them disassemble any game that used RTLink/Plus. Then no-one else will have to go through all the frustrations that I did trying to deal with this %#@! thing.

by Dreammaster (noreply@blogger.com) at November 04, 2015 06:32 PM

October 31, 2015

Paul Gilbert (Dreammaster)

Sherlock Testing Resounding Success

Hi everyone,

It seems like the testing of the Sherlock games has been a success. There were lots of bug reported, and all the ones reported so far have been fixed. The foreign language versions haven't all been fully tested yet, but hopefully now the immediate crashes with conversations and inventory in both Serrated Scalpel and Rose Tattoo foreign language versions have been fixed, and the rest of the games can be tested. I'd like to thank everyone who's tested so far for your efforts, and feel free to post any more bugs you come across. Though hopefully there won't be too many more to find :). And if one else has copies of either game, particularly different foreign versions, any other testers would be appreciated.

Right at the moment I'm at the sweet point where all the outstanding bugs for Sherlock have been resolved, so unless new ones come in, I can turn my attention to other things.

So, whats coming next for me? Several things:

Serrated Scalpel 3DO

Serrated Scalpel 3DO still isn't completable. There are some areas, such as the darts game to fix up, and there are still some differences in sprite positioning that would need to be accounted for. It's somewhat constrained by the fact that we don't have any original source for it, and I don't have any experience with reverse engineering 3DO games. It may simply be a case of do as best we can with hardcoded fixes as necessary.

The 3DO version also has a few missing things compared to the PC version; notably it lacks the journal the PC version has. A "nice to have" for the future would be to reintroduce it, so that the 3DO version could be the definitive version of the game, containing all the PC version functionality as well as the video for all conversations. We'd likely create a tool that extracts necessary graphics for UI buttons and the journal background from the PC version, and produce a Dat file that ScummVM can use automatically when playing the 3DO version.

RTLink Overlay manager

Next, there's the RTLink/Plus overly handling in Companions of Xanth. I'd previously had some luck writing a tool to process Rex Nebular and create a flat executable with all the segments suitable for disassembling, but doing the same for Companions of Xanth proved elusive. Over the years I made several attempts, but none bore fruit. Until now. As of yesterday, I was finally able to write a new version of the tool that successfully generated a flat executable that could be disassembled. So one day, after a great deal of disassembly work, ScummVM may support the game.

My only disappointment is that RTLink/Plus seems to have had quite a number of variations. Rex and Xanth's RTLink mechanism were fundamentally similar, with all the RTLink code, segment list, and method thunks/stubs in the executable and/or overlay file. For several other games with RTLink that I tried, however, they seem to use a bizarre alternate method where a file called 'rtlinkst.com' is loaded, then an '.RTL' file for the game, and finally only then the game executable. Which means that my tool doesn't work with them, and I'd need to figure out this new mechanism from scratch if I want the tool to be general-purpose enough to handle any RTLink game anyone might want to use it on in the future.

Guess that can be another long-term project to muck around with. Hopefully it won't take as long as it did to finally get Xanth properly disassembled. :). I'll probably make another posting in a day or so about RTLink in more detail, for those that are interested.

Might & Magic, World of Xeen

Next there's my work on re-implementing Might & Magic - World of Xeen (and Clouds of Xeen, Dark Side of Xeen, and Swords of Xeen). With some free time last weekend, I finally returned to working on them, and likewise was finally able to properly disassemble the  algorithm the original used for scaling. So as of now, sprites are now correctly scaled, as before I was only using a rough guess scaling code I'd nicked from another game engine. I've also fixed some other drawing bugs for drawing outdoor areas (outside town). So as of now the game scenes now display properly! :)



That's right, you can now walk around, fight monsters, go visit the various buildings in town, and even leave the town! In fact, most of the functionality for the games are already implemented. Apart from lots of testing and minor bugfixes that will be needed, only the following major areas remain to be implemented:

  • Introduction/ending sequences for the games
  • character management, and title screens.
  • Sound. I'm hoping I can simply slot in one of ScummVM's audio decoders without much further work.
  • Savegames

At the moment I'm concentrating on getting World of Xeen (which combines Might & Magic 4 and 5 together) working. But then afterwards I'll implement separate support for 4 and 5, as well as for Swords of Xeen. It might even be feasible to handle Might and Magic 3 - Isles of Terra as well, since I'm given to understand the engines are nearly the same.

Those interested in following the progress can see it at the brand-spanking new RogueVM Github account. That's right; after all the years of idle talk, I've finally set up a place to properly store RPG related game engines. No website yet, but at least it's a start. :) I'll likely spend the near-term focusing mostly on finishing support for the game before I move onto any other adventure games, considering how far along the engine is already.

Return of the Phantom

Strangerke has put a lot of work recently into implementing scene logic for Return of Phantom, the next MADS game that was published after Rex Nebular. There are quite a few stubs for missing engine functionality that was added in though. So when I do return to working on adventures, it will likely be to assist him in completing the game.

DreamMaster.

by Dreammaster (noreply@blogger.com) at October 31, 2015 04:01 AM

September 27, 2015

Matthew Hoops (Clone2727)

The Future

On the off-chance you didn't recognize the image from my last post, it was part of an animation from Buried in Time. And now you can find the source for running the game in ScummVM over here. And, yes, it's completable.

I'll fill in more details on the blog when I get a chance, but I think trying it out is probably a better way to show off what's been done.

Oh, and I went a little overboard with compatibility on this title. Not only is the 24-bit one working, it has full compatibility with the 8-bit game resources too. (This is the ultimate purpose for that Cinepak dithering I spoke of last year.)

Much thanks to Presto Studios yet again for making all this possible and providing the original source code!

by clone2727 (noreply@blogger.com) at September 27, 2015 12:45 AM

July 06, 2015

Paul Gilbert (Dreammaster)

Rose Tattoo is in progress

Hi all,

Work is progressing well on Rose Tattoo, and as of tonight I hit a major milestone.. the entire game intro sequence is now completable. See the screenshots below:



There are still some minor graphic glitches at various points, and one of the scenes which is a double-side scene isn't properly scrolled horizontally yet, but even so, it's a great step forward in supporting the game. We're actually lucky in that Rose Tattoo implemented all of the introduction using standard game scenes. So it saved a lot of effort implementing manual introduction code like had to be done for Scalpel.

Now that the introduction is working, more or less, I'll be devoting more time to implementing the game-play. I'd already been spending some of my working on it, so some interaction is possible.. you can look at objects, open up the inventory, and conversations with characters are partially working. The game map is also already implemented, so you can get to other game locations as well.

On another subject, I had good luck implementing the original EA logo at the start of Serrated Scalpel. I was able to complete support for it in the TsAGE engine, and then used that as a basis for copying necessary code into the Sherlock engine. With some most welcome assistance from others, the EA logo at the start of the game now displays when you start the game, just like the original does.

Finally, on yet another tack, there have been some promising first steps towards supporting the 3DO version of Serrated Scalpel by m-kiewitz, with some assistance from clone2727.  The 3DO was a superior version of game, and included 16-bit color, video portraits, and full speech for every conversation in the game. Supporting this would be great. It would be nice if, one day, the 3DO version could be re-released with ScummVM, so a wider audience could properly enjoy the game.

by Dreammaster (noreply@blogger.com) at July 06, 2015 04:09 AM

January 25, 2015

ResidualVM News Headlines

ResidualVM 0.2.1 - "Działające Napisy" released

Version 0.2.1 is a bugfix release for 0.2.0. It fixes a crash when using subtitles in the Polish version of Myst III, and an IRIX port was added.

As always, the latest release is available from our downloads page.

by bgK (nospam@residualvm.org) at January 25, 2015 12:00 AM

December 31, 2014

ResidualVM News Headlines

ResidualVM 0.2.0 - "Deliverance" released

We are proud to announce the release of a new version of ResidualVM, bringing Myst III: Exile support, more than thirteen years after its original release.

Initial work towards adding Myst III support began back in 2009 as a side project. A work in progress version of the engine was included in ResidualVM two years later. The game became completable in 2012, but only now do we feel confident enough to release a stable version providing a feature complete experience.

Additionally, some bugs have been fixed in Grim Fandango, and game data verification has been added on first launch. So that you'll know if your game data was copied correctly from your CDs.

Whether you are using Windows, Linux or OS X, you can check out our downloads page, get yourself a copy of ResidualVM 0.2.0, and decide later if you'd rather visit the land of the dead or the land of a revengeful trapped man. The changes in version 0.2.0 are detailed in our NEWS file. Instructions to setup the games can be found on our wiki.

As usual, even though we have tested the games thoroughly you might encounter game-breaking bugs. So, please save often and report bugs to our bug tracker.

Our thanks go to Presto Studios, for releasing such a great game, and to the people involved with testing ResidualVM over the years, allowing us to provide high quality releases.

2015 already looks like a promising year for ResidualVM, thanks to the ongoing effort for adding Escape from Monkey Island support.

by bgK (nospam@residualvm.org) at December 31, 2014 12:00 AM

December 23, 2014

ResidualVM News Headlines

Playtesters needed for testing of Grim Fandango and Myst III for 0.2.0 release

We are planning to release ResidualVM 0.2.0 with support for Myst III as our second supported game. The Grim engine has had quite a few changes since 0.1.1, so to avoid any big showstoppers for Grim Fandango, we need your help.

We need people to play through Grim Fandango and Myst III, preferably testing all the optional parts that aren't required for completion too. If you are interested, download our 0.2.0 pre-release builds (not the unstable builds), and start playing.

Any bugs should be reported to our issue-tracker on GitHub, and when you complete, you should post in our forums.

Details can be found here.

by aquadran (nospam@residualvm.org) at December 23, 2014 12:00 AM

November 30, 2014

Thierry Crozat (criezy)

Broken Speech

Yes, another Broken Sword post. In previous posts I wrote about my initial work to add support for the mac version of the Broken Sword game in ScummVM and some more work I did to fix graphical glitches with this version. At that point the game was working fine for me. But soon, we got a report on the forum that the speech was not working. Obviously it was working for me, I would have noticed if it wasn't. So what the heck?!

Bug reports are good. They stop me getting bored. And they show that other users have the mac version of Broken Sword and benefit from my work, which is also gratifying. So let's look at that issue. And to do so, first let's rewind to my first post. I wrote that I assumed the files with the same name (including the extension) as the files of the Windows version where in the same format, and in particular used the same endianness. And the files with a different extension were big endian in the mac version and little endian in the Windows version. That proved mostly correct (a few resources had been left as little endian data in files otherwise converted to big endian).

Except that wasn't correct. So why did it work? Because I had been lucky. When I initially worked on supporting that game it looked like my guess was correct, and I therefore made quick progress as I was not distracted by some strange behaviour. But...

Let's start the story from the beginning:

In the Windows version the speech is stored in a file named speech.clu. There are actually two such files, one on each CD, and they store the speech as 16 bits compressed mono wave data. And as you can expect the data is little endian.

In the Mac version, the files have the same name (speech.clu). So in my initial implementation I assumed the speech data was little endian in the mac version as well. And it worked... with the version I have (the French version). Obviously it didn't work with the version of the user reporting the bug (the English version) since the user reported hearing static noise instead of speech.

The two files (Windows and Mac versions, both English) have the same size:


But opening them in an hex editor shows differences:

Do the differences remind you of something?
If not go back and read again the first two posts in this Broken Sword mac support series.

Before looking at the differences, I will give a short explanation of the speech file format.
The speech files are a collection of sound resources. Each resource contains the wave data for spoken sentence and is organised as follow:
4 bytes: 'data' (i.e. hexadecimal values 64 61 74 61)
4 bytes: number of samples
n bytes: wave data (16 bits mono)

So quite simple, but maybe no as simple as you might think. If you are thinking that n is the number of samples multiplied by 2 (since each sample takes 2 bytes) you are wrong. Because the data is compressed. This is not really important for now so I will keep the description of the compression for later.

What is important here is that we can see that the first 4 bytes after 'data' are identical (in the image above hexadecimal values 8E E6 01 00 - which, since we know it is little endian, means 0x0001E68E = 124558 samples). But the values that follow are obviously a series of 2 bytes values for which the bytes have been swapped.

At this points, here is a small reminder in case you are not following me and did not go back to my previous posts: big endian and little endian are conventions used to interpret the bytes making up a data word (more at http://en.wikipedia.org/wiki/Endianness).  For example 42 in hexa is 2A, or when using two bytes 002A. When using the big endian convention, this would be stored as 00 2A. But when using the little endian convention this would be stored as 2A 00.

So this should be obvious to you now that both the Windows and the Mac version store the number of samples of the sound resource as little endian values, but in the mac version the data samples themselves are stored using big endian convention. Why mix endianness in the same file? Why do this for the mac English version but not the mac French version? Don't ask me, I have no idea.

Since some mac version use little endian and others use big endian, we need to know which one it is. Does it depend on the language, i.e. all French versions use little endian and all English versions use big endian? Maybe. But I don't trust statistics on a set of two samples. And what of the German versions?

Therefore we decided to use a heuristic to find out if the mac version the player has uses big endian data or little endian data. The heuristic works by computing the average difference between two consecutive samples (using absolute values). Using the assumption that a sound wave is smoother than picking values at random, the lower average difference is considered to be the correct endianness.

If we take the 13 samples from the example image above, assuming big endian for the mac version gives us the following curve:


The average difference value from one sample to the next is 425.17.

If we assume little endian data the curve is:

And the average difference value is 9344.75.

So in this case the heuristic tells us the data is big endian, which it is. Of course in the actual source code we use more than 13 samples to get a statistically valid heuristic value.
The original patch that adds the heuristic code can be found in the patch tracker: https://sourceforge.net/p/scummvm/patches/956/

But the story does not ends here. A new bug report very similar to the original one (i.e. speech sounds like static noise) was reported a few months ago. It was quite obvious that the heuristic did not work for that user and the wrong endianness was used. Why is that? It turns out there were several issues with the original heuristic code.

And that is where explaining how the speech data compression works will help to understand what was wrong. The data for one sound resource is broken in blocks, each one starting with a number of samples followed by the sample values. When you have consecutive samples with the same value it uses a negative size and the value is stored only once.
So for example the following sequence:

    0 0 0 0 0 1 2 3 4 5

would be stored as (where the brackets indicate the blocks):
    [
-5 0] [5 1 2 3 4 5]
All those numbers (number of samples and sample value) are stored on 2 bytes.

Here is the original source code (if you don't see the source code visit the blog as it may not be visible in RSS feeds).

I will not show the uncompressSpeech() code (yet). The only thing you need to know is that it uses the value of _bigEndianSpeech as either big endian or little endian data. So what the code above does is get the data assuming little endian data and then compute the heuristic value for the samples it gets and for the same samples to which a byte swap is applied, which should be the value we would get assuming big endian data. Right?

Wrong! This heuristic forgets something: the data is compressed, and when uncompressing it always assume little endian when reading the number of samples for each block. But if the data are big endian this number would be different and the blocks would have different sizes, and because the block boundaries would be wrong it would cause number of samples to be interpreted as sound samples and some sound samples to be interpreted as number of samples.

For example let's look at the resource of 10 sample with the compressed data -5 0 5 1 2 3 4 5.
Assuming the data is stored in big endian, in hexadecimal values with two bytes per value this give us: 80 05 00 00 00 05 00 01 00 02 00 03 00 04 00 05
If we read this with the heuristic above, because the number of sample is always read assuming little endian data we get 80 05 = 1408 samples instead of -5 for the first block. So the code will get the following samples: 0 5 1 2 3 4 5 followed by 4 garbage values read beyond the end of the resource instead of getting 0 0 0 0 0 1 2 3 4 5. So if the data are stored using big endian, the heuristic values gets biased. Almost always it still gets a lower score as reading it as little endian though.

The solution here is to call uncompressSpeech() twice, once assuming little endian and once assuming big endian.

But there is still an issue with this heuristic.
When reading with the wrong endianness, since we may read the wrong length, it may for example be a big negative number. Because we are using a relatively small finite number of samples, statistically we could end up with a small heuristic value because it has a lot of consecutive samples with the same value. For this reason I made an additional change to skip consecutive samples with the same value when computing the average difference.
After that commit the value for the heuristic with the wrong endianness is consistently about 21000, i.e. 1/3rd of 16 bits integer range (65 535 / 3 = 21845). As noted by wjp:  the average absolute difference between two random numbers drawn independently from a uniform distribution between 0 and N is indeed N/3. So this is quite reassuring.

So everything was correct after this change? No, that would be too easy. We want complex puzzles spanning several rooms, not some kind of hidden objects game. And the user reporting the bugs confirmed the bug was still present after that change. So let's look at a different room, or rather a different function.

Here is the code from uncompressSpeech():
Note something relevant? No? Look closer. You see it now? Yes, this function always give us the sound data in little endian format, whatever the endianness in which it is stored, and more importantly whatever the endianness of the computer on which the code is run.

And if you look at the heuristic code above, it assumes it gets data in the native endianness (i.e. the endianness of the computer on which the code is run). So when running on a computer using big endian convention the heuristic was wrong. Let's look again at our previous example and how it was interpret on a big endian computer:
   Read with the correct endianess: 0 1 2 3 4 5 (duplicate values have been removed)
   Was interpreted as: 0 256 512 768 1024 1280

   Read with the incorrect endianess: 0 1280 256 512 768 1024 1280
   Was interpreted as 0 5 1 2 3 4

So in the heuristic code, on a big endian computer we need to swap the bytes of the two set of data to get the correct value. This brings us to the final code, in which the heuristic computation was also moved to a separate function to avoid code duplication (since it is done twice):

The user reporting the bug confirmed he was using ScummVM on a big endian computer (a G4 mac) and that the speech was correct after that final change.

by Thierry Crozat (noreply@blogger.com) at November 30, 2014 09:38 PM

November 29, 2014

Sven Hesse (DrMcCoy)

xoreos Not-Thanksgiving

I elaborate on the current state of the xoreos project and what/who I am thankful for in this new post on the xoreos blog.

flattr this!

by DrMcCoy at November 29, 2014 03:31 PM

November 23, 2014

James Woodcock

My ScummVM Music Enhancement Project & ScummVM feature in Gamer.no Article

Screen-Shot-2014-11-23-at-10.02.41-1100x518
My attempts to recreate classic point and click adventure game soundtracks on modern hardware and have them playable within the actual games through ScummVM known as the ‘ScummVM Music Enhancement Project‘ has featured in one of Gamer.no’s articles by Erik-André Vik Mamen recently.

The article was originally written in Norwegian, but luckily Google Translate to the rescue…

It is also another person who shall have praised for having changed the gaming experience of old classics. It is James Woodcock, through his “ScummVM Music Enhancement Project”.
Gamer.no ScummVM Article (English translation via Google Translate)

The article continues…

It took considerably less time to get permission from other developers and publishers. Beneath a Steel Sky and Discworld followed quickly afterwards. To get a personal response from Terry Pratchett (author of the Discworld books, ed.’s Note.) Was clearly a fantastic feeling.
Gamer.no ScummVM Article (English translation via Google Translate)

Make sure you read the entire article on ScummVM.

//www.youtube.com/watch?v=ZsJ0iutFCvA

by James Woodcock at November 23, 2014 10:18 AM

 

curved edge   curved edge