Welcome to the ScummVM planet - This aggregates the personal blogs of developers, teams members and active participants from all around the ScummVM community.
If you wish to subscribe to updates to the planet or individual blogs please use the links on the right hand side.
To add your blog to the planet contact DJWillis.
November 27, 2015
xoreos is a FLOSS project aiming to reimplement BioWare’s Aurora engine (and derivatives), covering their games starting with Neverwinter Nights and potentially up to Dragon Age II. This post gives a short update on the current progress.
Note: This is a cross-post of a news item on the xoreos website.
The end of the year is approaching fast, and just like last year, I want to use this time for some retrospection.
First of all, what happened in the last year?
- berenm added support for building xoreos with CMake, by the way of parsing the automake files used for the autotools build system. This way, xoreos can now be built with either CMake or autotools. I was skeptical at first, especially since I harbour no love for CMake, but it is working reasonably well and I am quite happy with it. In hindsight, I was wrong to reject this pull request for so long.
- I focused on supporting all the different model formats used in the Aurora games, and then I made all the games display their in-game areas with objects.
- xoreos adopted the Contributor Covenant as its Code of Conduct, in the hopes that it helps foster a friendly and welcoming community.
- The big one: our first official release, xoreos 0.0.2, nicknamed “Aribeth”.
- I overhauled the script system, making it more generic. This way, I was able to apply it to all targeted games, except Sonic Chronicles: The Dark Brotherhood (which doesn’t seem to use any scripts at all). This included figuring out and implementing four new script bytecode opcodes: two for array access in Dragon Age: Origins, and two for reference creation in Dragon Age II.
- I implemented reflective environment mapping for Neverwinter Nights and the two Knights of the Old Republic games.
- I added a new tool to the xoreos-tools package: xml2tlk, which can recreate TLK talk table files out of XML files created by tlk2xml.
- With these changes, I decided to push out xoreos 0.0.3, nicknamed “Bastila”.
This is all old news, more or less already discussed in previous blog posts. However, since then, I added yet another new tool to the xoreos-tools package: ncsdis. It’s a disassembler for NCS files, the stack-based compiled bytecode of the C-like NWScript, BioWare’s scripting language used throughout their Aurora-based games.
It basically replaces the disassembler within the old OpenKnightsN WScript compiler, with various added benefits. I’ll write a bit more about this tool in the near future, so for now I’ll just leave you with an example assembly listing it can produce, as well as a control flow graph it can create (with the help of Graphviz). As you can see, it already groups the instruction by blocks and subroutines. It performs a static analysis of the stack (to figure out subroutine parameters and return types) and it also analyzes the control flow to detect assorted control structures (loops, if/else). I plan to grow it into a full-fledged NWScript decompiler.
Additionally, I also added support for BioWare’s Neverwinter Nights premium modules, like Kingmaker, to xoreos.
On the documentation side of things,
- I added comments and documentation to various files in the xoreos sources, hopefully making them more understandable and useful for potential new contributors and otherwise interested people. Considering how awful my memory is at, this is also a kind of future-proofing.
- Farmboy0 added “research” subpages for various games on our wiki, filling them with information about their workings.
- I extended our TODO list considerably.
- I added an example configuration file, and extended the documentation on the wiki on how to compile and run xoreos.
- I wrote man pages for each tool in xoreos and for xoreos itself. I also added the former to the wiki.
Phew! This is again a bigger list than I had anticipated. This wouldn’t have been possible without these people, for whom I am thankful:
- I am thankful to berenm for providing the CMake bindings, despite my grumbling about it.
- I am thankful to Supermanu, for continuing on chipping away on the Neverwinter Nights character generator.
- I am thankful to Farmboy0, for working on xoreos’ Jade Empire engine and researching game internals.
- I am thankful to mirv, for continuing with the huge task of rewriting my naive OpenGL code.
- I am thankful to Coraline Ada Ehmke for creating the Contributor Covenant.
- I am thankful to all the people in the different BioWare modding communities, for having figured out many different things already. Skywing for example, who had emailed me a few years ago about certain NWScript issues, issues I recently stumbled over again.
- I am thankful to fuzzie, for giving me pointers on the NCS disassembler/decompiler.
- I am thankful to the GamingOnLinux people, who do a lot of work reporting on all sorts of Linux-related gaming news, and who so graciously mirror my xoreos blog posts.
- I am thankful to kevL, for notifying me of issues with xoreos’ build system on configurations I hadn’t thought about.
- I am thankful to clone2727, for putting up with rants and ravings.
- I am thankful to all the people who told me when I was wrong, for example when I wrongheadedly silenced clang static analyzer warnings, without understanding what I was doing.
- I am thankful to everybody else who gave me hints and tips, taught me tricks and procedure, showed me new things, old things, forgotten things, broken things.
- I am thankful to all the people who are not angry with me for forgetting them, because they are aware that this is not meant as a personal slight ;).
Now that I have these mushy feelings out of my system, here’s hoping for another great year!
And like always, if you want to join our effort, please don’t hesitate to contact us!
by DrMcCoy at November 27, 2015 02:26 AM
November 22, 2015
November 11, 2015
November 04, 2015
Finally, after many years of half-hearted attempts, I've finally rewritten my RTLink decode tool practically from scratch to handle disassembling the RTLink/Plus overlay management used by the later Legend entertainment games. See rtlink_decode
for the result. The original version of the tool was pretty dodgy, and pretty much hardcoded to only handle the MADS games (Rex Nebular, Return of the Phantom, and Dragonsphere). In this posting, I'll go into more detail of what RTLink/Plus was for those who may be interested.
In the latter days of DOS gaming, game developers started running into a problem. Namely, that their executables were starting to hit the limits of main memory. Not every game could, or would, take advantage of scripting game content, so having all the game logic in the main executable caused the size to bloat. So what to do if your executable was now too big to fit in memory? This was the problem RTLink/Plus was designed to solve.
RTLink is essentially an overlay manager. It splits a program's compiled code into multiple different segments, and allows them to be loaded in as needed, and then replaced when code in other segments needs to execute. RTLink can handle recursive segments, with individual segments split up into their own set of swappable sub-segments. It also allows for multiple different "loading areas" in memory that can independently have their own set of segments. See this article
for more general information about RTLink.
Before I go into more details of the problem this posed for disassembly, let's go over how RTLink implements the overlay manager in code. So far, I've encountered three different variations on RTLink being used in executables. What I'll call variation 1 & 2 seem to be the most common form of RTLink in games I've examined. When a program is compiled with either of these versions of RTLink/Plus, one of the segments in the code will contain the RTLink logic, as well as two main areas: the dynamic segments and the function thunks.
The segment list is a list of the dynamic segments within the application. It contains the following information:
- The segment in memory where the dynamic segment should be loaded
- Whether the segment is stored in the application or a secondary overlay file (Variation 1 only, version 2 only ever uses the executable).
- The file offset and size of the segment
- The number of relocation entries the segment has; Variation 1 only. Variation 2 has it as part of the starting header for the segment pointed to.
When a segment is needed, the above details are used to read the segment's data from file, and load it into the correct place in memory. The data for a segment consists of two parts: an initial header area, and the date/code for the segment. For variation 1, the header area simply consists of a list of relocation entries. Whereas for variation 2, the details of segment size and number of relocations are provided in a header at the start of the segment, before the relocations list.
A segment's relocation entries are used for the same purpose as relocation entries in a standard application - executables can be loaded at different locations within memory, so all segment references need to be relative to the starting point of where the program is loaded. By keeping the relocation entries for each dynamic segment together with the segment data itself, it's easier for RTLink to apply any needed segment adjustments each time a dynamic segment is loaded.
This is fine to handle shifting the segments in and out of memory, and allow them to have valid memory references, but what causes them to be loaded? The answer is the method "thunks" area of the RTLink segment. When dealing with dynamic segments, you can't just do a far call to some offset in the area of memory segments are loaded in.. you couldn't be sure that the segment you want is actually loaded, or still in memory and not unloaded by some other segment. For this purpose, the thunk list is present.
For every method in a dynamic segment that is referenced by any other segment, a thunk/stub method is created. These consist essentially of the following: a call to the RTLink manager to load the correct segment for the method, a far jump to the method in the correct memory location in the loaded segment, and a following 16-bit value specifying which segment the thunk is for. This way, the thunk method acts as a wrapper, ensuring the correct segment is loaded and passing control to the method to execute.
For variations 1 and 2, the thunk methods have some minor differences, such as version 2 using far calls to the RTLink segment loading code, and having an optional word after the segment index. The segment selector in the far jump call is also already loaded with the memory segment in variation 1, whereas in version 2 it's normally 0 initially, and then set to the correct segment when the thunk method is called. This allows variation 2 to dynamically load the segment in different places in memory, whereas variation 1 is limited to a single specific loading point.
The RTLink segment loader method also mucks around with the stack to push a new intermediate return address on the stack for when the method that's jumped to finishes. This return address points to a code fragment that also handles the case where a method in a dynamic segment calls a method in another one.. in that case, it handles reloading the original segment, so that the original caller's code can be safely returned to.
Put altogether, this scheme allows programs of practically any size needed. As the program grows, the code simply needs to be split into more and more dynamic segments which will get loaded only when needed, and remain on disk when not. Great for having big programs, but not so great for those of us interested in reverse engineering the game by disassembling the executable.
There were several problems to be solved for disassembling such games, which I'll go into now.
A standard IDA disassembly doesn't have all the code
Well, it wouldn't. If you try to disassemble an RTLink/Plus compiled game, IDA will give you an error about unused data at the end of the executable. This will be for one or more RTLink segments. Additionally, as previously mentioned, some of the code for the program can also be stored in a separate OVL (Overlay) file.
Well, I could just load the raw data for them into the disassembly, right?
Well, no. That wouldn't help much, because of all the thunk methods. They all have their references to the same area of memory where segments are expected to be loaded. If you were doing things manually, you'd need to get the details of each segment from RTLink, manually load the code and/or data into new IDA segments, and then manually adjust the thunk methods to point to those methods.
You'd also need to worry about the dynamic segment relocation entries. If you manually loaded the code for a segment, you'd have to read the list of relocation entries for the dynamic segment and manually adjust each relocation entry within the segment. Segment selectors may point to code within the segment (or another sub-segment within the loaded overall RTLink segment), to a low memory area of the executable that remains static in memory, or to the data segment (at a higher memory segment). All in all, you'd have to be extraordinarily patient to all that by hand.
So that's why you wrote rtlink_decode, right? That's what it does?
Yes and no. A bit part of what it does is indeed doing the above to create a new executable suitable for disassembly. This includes laying out all the dynamic segments sequentially (without their segment headers and relocation lists), handling relocation fixups, and the thunk methods adjusted to point to their methods in the decoded executable. However, another problem crops up in the handling of the data segment.
In my experience with RTLink, I've come across across two types of data segments:
- In the case of the later Legend Entertainment games, the executable has a single RTLink segment, with the remainder of the segments coming from an OVL file. The single executable segment is for the main data segment as well as a few other miscellaneous segments.
- In the case of the MADS games, the data segment isn't an RTLink segment, but all the RTLink segments follow it in the executable.
In both cases, we have a problem doing a proper disassembly. Executables are normally expected to have the data segment at the end of the program, because the data segment may be longer than the end of the executable. For example, a game's data segment may only have 1Kb of pre-set values which are stored in the executable, but it still requires 40Kb of unallocated/uninitialized space. That's why you'll frequently see, when you do a disassembly of a program, areas at the end of a data segment with '?' mark values, indicating the memory isn't part of the executable, so doesn't have any specific value when the program starts.
So if we did just lay the segments end to end, the data segment, coming before other dynamic segments, would end up being shorter than it should be, and a lot of the references to data within it would end up wrapping onto the following dynamic segments in the reworked executable. To avoid this, the rtlink_decode tool ensures that the data segment falls at the end of the generated executable, after all the other segments. This, however, causes it's own share of problems. All the existing references to the data segment refer to where the data segment was expected to be loaded in memory, not to where the data segment actually is in the new executable. Because of this, all the references to the data segment in the executable have to be adjusted accordingly.
Ouch! Sounds fiddly.
It is. And took a lot of messing around to get right. Even then, that's not the entirety of the picture. For Companions of Xanth, the Legend game I used for testing when rebuilding the tool, the data segment has some extra gotcha's.. It contains segment references into the middle of the memory area RTLink segments are loaded into. Presumably these are used in some special controlled circumstances when a specific segment (or segments) are loaded to access particular data. But it's impossible to know without understanding the game a lot better.
Worse, the presence of the references were screwing up some of the loaded dynamic segments in the disassembly, causing them to be split in half. To handle this, the tool explicitly looks for such "bad" references in the data segment, and removes the relocation entries for them. This way, the value in the data segment will remain as a static word, and the segments don't get incorrectly split up. The user can always then later manually set up a pointer to an appropriate segment if they wish. This handles the bulk of such errors, but Xanth at least, there are still references in the low part of the executable (that remains static in memory) to locations within the RTLink segments. Since I can't know which particular RTLink segment is meant to be loaded when the code they're in is called, these few remaining references will have to be later manually adjusted as well.
So that's it?
Yep. After all these years, I'm finally able to generate a (mostly valid) "decoded" executable, and produce a clean disassembly of Companions of Xanth. I also, initially, had two separate versions of the the tool, one the old hacky version for MADS games, and the legend variation for Legend-style RTLink usage. I've since updated my tool to properly handle MADS games, so now there's only the single rtlink_decode tool, and it can handle both variations 1 and 2.
Oh, wait.. what about the 3rd variation you mentioned?
Ah, yes, I didn't really get into that, did I. This version seems to be somewhat different than the other two variations. In this case, the RTLink code is stored in a separate rtlinkst.com file, and then loaded into memory. It then shifts part of the program downwards in memory, and uses it's own relocation table to manually process relocation entries on the shifted code. This variation is proving tricky to disassemble, so whilst I have located the segment list, I still need to:
- Figure out how relocation data is encoded. I think I've located the correct data in the executable, but the code RTLink uses to update relocation entries is pretty nasty and overcomplicated.
- How much of the start of the executable to remove so that the produced executable doesn't have any of the old code at the start of the executable that gets overwritten
- Find the thunk methods, and see whether the existing code will handle them.
Hopefully I can quickly figure out the remaining details for the third variation soon. The goal is to have a tool that both myself and others can use in the future to help them disassemble any game that used RTLink/Plus. Then no-one else will have to go through all the frustrations that I did trying to deal with this %#@! thing.
by Dreammaster (firstname.lastname@example.org) at November 04, 2015 06:32 PM
October 31, 2015
It seems like the testing of the Sherlock games has been a success. There were lots of bug reported, and all the ones reported so far have been fixed. The foreign language versions haven't all been fully tested yet, but hopefully now the immediate crashes with conversations and inventory in both Serrated Scalpel and Rose Tattoo foreign language versions have been fixed, and the rest of the games can be tested. I'd like to thank everyone who's tested so far for your efforts, and feel free to post any more bugs you come across. Though hopefully there won't be too many more to find :). And if one else has copies of either game, particularly different foreign versions, any other testers would be appreciated.
Right at the moment I'm at the sweet point where all the outstanding bugs for Sherlock have been resolved, so unless new ones come in, I can turn my attention to other things.
So, whats coming next for me? Several things:Serrated Scalpel 3DO
Serrated Scalpel 3DO still isn't completable. There are some areas, such as the darts game to fix up, and there are still some differences in sprite positioning that would need to be accounted for. It's somewhat constrained by the fact that we don't have any original source for it, and I don't have any experience with reverse engineering 3DO games. It may simply be a case of do as best we can with hardcoded fixes as necessary.
The 3DO version also has a few missing things compared to the PC version; notably it lacks the journal the PC version has. A "nice to have" for the future would be to reintroduce it, so that the 3DO version could be the definitive version of the game, containing all the PC version functionality as well as the video for all conversations. We'd likely create a tool that extracts necessary graphics for UI buttons and the journal background from the PC version, and produce a Dat file that ScummVM can use automatically when playing the 3DO version.RTLink Overlay manager
Next, there's the RTLink/Plus overly handling in Companions of Xanth. I'd previously had some luck writing a tool to process Rex Nebular and create a flat executable with all the segments suitable for disassembling, but doing the same for Companions of Xanth proved elusive. Over the years I made several attempts, but none bore fruit. Until now. As of yesterday, I was finally able to write a new version of the tool that successfully generated a flat executable that could be disassembled. So one day, after a great deal of disassembly work, ScummVM may support the game.
My only disappointment is that RTLink/Plus seems to have had quite a number of variations. Rex and Xanth's RTLink mechanism were fundamentally similar, with all the RTLink code, segment list, and method thunks/stubs in the executable and/or overlay file. For several other games with RTLink that I tried, however, they seem to use a bizarre alternate method where a file called 'rtlinkst.com' is loaded, then an '.RTL' file for the game, and finally only then the game executable. Which means that my tool doesn't work with them, and I'd need to figure out this new mechanism from scratch if I want the tool to be general-purpose enough to handle any RTLink game anyone might want to use it on in the future.
Guess that can be another long-term project to muck around with. Hopefully it won't take as long as it did to finally get Xanth properly disassembled. :). I'll probably make another posting in a day or so about RTLink in more detail, for those that are interested.Might & Magic, World of Xeen
Next there's my work on re-implementing Might & Magic - World of Xeen (and Clouds of Xeen, Dark Side of Xeen, and Swords of Xeen). With some free time last weekend, I finally returned to working on them, and likewise was finally able to properly disassemble the algorithm the original used for scaling. So as of now, sprites are now correctly scaled, as before I was only using a rough guess scaling code I'd nicked from another game engine. I've also fixed some other drawing bugs for drawing outdoor areas (outside town). So as of now the game scenes now display properly! :)
That's right, you can now walk around, fight monsters, go visit the various buildings in town, and even leave the town! In fact, most of the functionality for the games are already implemented. Apart from lots of testing and minor bugfixes that will be needed, only the following major areas remain to be implemented:
- Introduction/ending sequences for the games
- character management, and title screens.
- Sound. I'm hoping I can simply slot in one of ScummVM's audio decoders without much further work.
At the moment I'm concentrating on getting World of Xeen (which combines Might & Magic 4 and 5 together) working. But then afterwards I'll implement separate support for 4 and 5, as well as for Swords of Xeen. It might even be feasible to handle Might and Magic 3 - Isles of Terra as well, since I'm given to understand the engines are nearly the same.
Those interested in following the progress can see it at the brand-spanking new RogueVM
Github account. That's right; after all the years of idle talk, I've finally set up a place to properly store RPG related game engines. No website yet, but at least it's a start. :) I'll likely spend the near-term focusing mostly on finishing support for the game before I move onto any other adventure games, considering how far along the engine is already.Return of the Phantom
Strangerke has put a lot of work recently into implementing scene logic for Return of Phantom, the next MADS game that was published after Rex Nebular. There are quite a few stubs for missing engine functionality that was added in though. So when I do return to working on adventures, it will likely be to assist him in completing the game.
by Dreammaster (email@example.com) at October 31, 2015 04:01 AM
October 22, 2015
Welcome to the third post in the series Technical Art!
We have been introduced to the concept of scripting in Maya and so far it seems very straightforward.
The thing that fascinates me most is that Maya seems to use its internal language to handle pretty much every action that the user can perform in the tool: this means that scripting has limitless potential to automate any kind of action and can manipulate the scene with a lot of freedom, as it has effectively the same scope of action that a normal user would have.
I have been experimenting with user interface creation scripts and combining those with the tasks that we have been given during the lecture, for example I expanded upon the first "cube army" script to make it configurable through a window.
I am pretty confident that I will be able to create a good user interface now that I know what kind of approach I have to use, and the documentation that Maya provides is very useful and it helped me a lot in finding the functions that I needed to achieve the result I wanted.
I will focus more on rigging in the next weeks as I want to have a more clear idea of the script I will be making for my coursework, but at the moment I still lack knowledge of some steps regarding advanced rigging tools that I think we will be covering in the next weeks, for now I will keep on practising and researching more advanced techniques, so stay tuned for more updates!
by Subr3v (firstname.lastname@example.org) at October 22, 2015 11:14 AM
Welcome to the second post in the series Technical Art!
During the past lectures we analysed the differences between rigs for animations and visual effects and rigs for games. As expected, games have more limitations due to the performance requirements, but this also varies in different types of games: mobile games will have more restrictions than a console game; but even within the same game some characters might have more resources allocated to them which means that the artist will be able to build a more complex rig for them (to achieve a more realistic animation).
We've also been introduced to the concept of skinning which is very crucial in determining how the model should be structured in terms of topology to have a visually pleasing deformation.
Since we are required to build a rig as part of our coursework I was researching models that I could use and as I don't have very strong modelling skills I started looking into already rigged characters: this way I can just remove the rig from the model and start from scratch.
I've always been fascinated by animal characters, so I started exploring this model and its topology seems to be very nicely laid out so I will be experimenting with it in the next weeks to see what kind of rig I can build with it.
The model should be reasonably complex for a game rig, it only has 4k triangles in total, the only problem I would see is that it might require too many joints for the head to make it look expressive enough, but one could just assume that if there are no camera close ups on the creature then such detail might not be required.
I will also be exploring more the scripting side of Maya in the next weeks, so stay tuned for more updates!
by Subr3v (email@example.com) at October 22, 2015 11:03 AM
Hi everyone, from today I will be starting a new post series about a module I am currently studying at university: Technical Art Applications.
Since the first lecture the module seems to be very interesting and I've always wanted to have a chance to have a better understanding on how 3D animators work and what the rigging process looks like, so choosing this module was a no brainer!
The most interesting thing about the module so far is that we have been given a lot of freedom on what to do for our coursework, which also means that I will need to do a lot of research on my own to understand what is best for me and to deliver a good piece of coursework by December.
I have decided to explore game rigs rather than animation or visual effects one simply because my main goal is to work for the games industry and I think that knowing more about how rigging works specifically in this field will be very valuable in terms of understanding how the artists work and what kind of technical problems they might have (and how I can help in solving those problems, obviously!).
Maya seems a very powerful tool and from what I've heard it supports either C++, Mel or python for scripting but we will be using python in the module so this will be a good opportunity to refine my Python skills and knowledge of Maya overall.
Even though it might be too early I have decided to put a lot of focus on the scripting side on for my coursework, but at the the moment I am still trying to figure out what kind of steps and processes could be automated and how far I can push the complexity of the script in this regard, so stay tuned for the next posts!
by Subr3v (firstname.lastname@example.org) at October 22, 2015 09:58 AM
September 27, 2015
On the off-chance you didn't recognize the image from my last post
, it was part of an animation from Buried in Time
. And now you can find the source for running the game in ScummVM over here
. And, yes, it's completable.
I'll fill in more details on the blog when I get a chance, but I think trying it out is probably a better way to show off what's been done.
Oh, and I went a little overboard with compatibility on this title. Not only is the 24-bit one working, it has full compatibility with the 8-bit game resources too. (This is the ultimate purpose for that Cinepak dithering
I spoke of last year.)
Much thanks to Presto Studios yet again for making all this possible and providing the original source code!
by clone2727 (email@example.com) at September 27, 2015 12:45 AM
September 25, 2015
September 13, 2015
Sherlock Holmes, greatest detective of all times, has a pair of puzzling cases to solve. Firstly, a gruesome murder outside a theatre. Police are clueless. Could it be the work of the notorious killer, Jack the Ripper? Then, in his second case, his brother Mycroft is nearly killed in an explosion. In this case, you'll play initially as Watson, who tries to snap Sherlock out of his despair, and then as Sherlock, as he tries to piece together why Mycroft was attacked, and why he called for him just prior to the explosion.
The ScummVM Team is proud to announce support for both games of the The Lost Files of Sherlock Holmes.. The Case of the Serrated Scalpel, and The Rose Tattoo. Many thanks go to Electronic Arts, for allowing us access to the original sources. Both games are now playable in ScummVM using the latest daily builds and ready for testing. For Serrated Scalpel, currently only the PC version is supported.. the 3DO version is mostly playable, but still needs further work. As usual, all bugs should be reported to our bug tracker following our bug submission guidelines. While you play through the game, we would also love it if you could take some screenshots for us.
So, have a play, and see if you're up to the challenge of stepping into the shoes of the great Sherlock Holmes. The game is afoot. Or in this case, games :)
by Paul Gilbert (firstname.lastname@example.org) at September 13, 2015 12:00 AM
July 06, 2015
Work is progressing well on Rose Tattoo, and as of tonight I hit a major milestone.. the entire game intro sequence is now completable. See the screenshots below:
There are still some minor graphic glitches at various points, and one of the scenes which is a double-side scene isn't properly scrolled horizontally yet, but even so, it's a great step forward in supporting the game. We're actually lucky in that Rose Tattoo implemented all of the introduction using standard game scenes. So it saved a lot of effort implementing manual introduction code like had to be done for Scalpel.
Now that the introduction is working, more or less, I'll be devoting more time to implementing the game-play. I'd already been spending some of my working on it, so some interaction is possible.. you can look at objects, open up the inventory, and conversations with characters are partially working. The game map is also already implemented, so you can get to other game locations as well.
On another subject, I had good luck implementing the original EA logo at the start of Serrated Scalpel. I was able to complete support for it in the TsAGE engine, and then used that as a basis for copying necessary code into the Sherlock engine. With some most welcome assistance from others, the EA logo at the start of the game now displays when you start the game, just like the original does.
Finally, on yet another tack, there have been some promising first steps towards supporting the 3DO version of Serrated Scalpel by m-kiewitz, with some assistance from clone2727. The 3DO was a superior version of game, and included 16-bit color, video portraits, and full speech for every conversation in the game. Supporting this would be great. It would be nice if, one day, the 3DO version could be re-released with ScummVM, so a wider audience could properly enjoy the game.
by Dreammaster (email@example.com) at July 06, 2015 04:09 AM
May 22, 2015
Looking back it seems, to my chagrin, that it's now been over a year since my last news posting was over a year ago. Whoops :P. Not that I haven't been keeping busy over the last year or so, with the release of Voyeur, Amazon - Guardians of Eden, Rex Nebular (finally), and of course, the newest game.. The Lost Files of Sherlock Holmes: Case of the Serrated Scalpel.
Many thanks go to EA for providing us access to the original source for this game. Also to forum user sirlemonhead, and to James Ferguson, who patiently over the last few years tried to make this happen. I've always been a big Sherlock Holmes fan, so it was fun to work on this project. It feels fitting to merge the game into master on the 22nd May, which is the birthday of the character's creator, Sir Arthur Conan Doyle. The engine isn't quite ready for serious testing yet.. it's still missing music playback, and there's also a starting logo animation that's not present. It should be finished soon, though, so expect to see an official testing announcement in the near future.
So.. as things stand, what I am up to right now?The Logo
Apart from the game proper, one of the more interesting things about the game is at the very beginning, where the publisher EA logo is shown. This logo display was actually implemented in a separate executable using the TsAGE engine, of all things. Original source for this couldn't be located, so it means that I'm having to reverse engineer it. Luckily, since we've already had experience with several other TsAGE titles, I've been able to make excellent progress in figuring out all the various TsAGE classes and their methods within the executable.
At the current point in time, I've identified the bulk of the core TsAGE classes, and the custom logic for the "game", which is contained a single scene class. This scene class consists of several scene objects, a few palette containers, and an "Action" class for coordinating what happens in the logo display. There's only a minor variation in how object sprites are loaded compared to the games that I still need to figure out.
I've already started implementing a new sub-module within TsAGE for the game logo. Once I finish that, it will be easier to analyse all the movement and frame changes of the images with the scene. Hopefully, based on that, I'll be able to simulate a similar sequence in our new Sherlock engine using the bare necessities from TsAGE - likely just the RLB archive manager and sprite loader. Particularly given the thoughtfulness of EA in providing us access to the original source, it would be nice to give them (the company) proper attribution by showing their logo just like the original game does.The Sequel
Apart from that, we have also been given access to source for the sequel, The Case of the Rose Tattoo. Implementing this is likely to be much more challenging, as the sequel changed over to a 640x480 display, and significantly altered the user interface. As such, it's likely it will need a lot of re-factoring of the code base to add support for it to the existing engine. If you thought a lot of re-factoring was done during the pull request, you 'aint seen nothing yet. :)
I'm also somewhat constrained by the fact that the original uses DOS4GW and a 32-bit code segment. Whilst we do have the original source, I need to be able to run the game in DosBox so I can actually see the code running, and check registers and memory contents at given points in the program. I've had some significant trouble with the DosBox debugger, trying to set breakpoints in the code so I can inspect the game's state. Doing so crashes either crashes DosBox, or the game executable, or the breakpoints simply aren't hit.
So far, I've only done some preliminary loading of scene resources in the second game, and the lack of a way to display the program state meant that I had to take a more laborious route of poring over the various resource structures and scene loading code in both games, to try and figure out what the differences were between the two, so my code can support it. Likely, as I proceed with implementing more of the game, this will cause real issues that will make finding bugs a lot harder.World of Xeen
It's been somewhat on a back-burner since I started work on the Sherlock Holmes games, but I had previously made real progress on re-implementing World of Xeen using the ScummVM framework. See below:
As you can see, I have much of the game interface implemented. You can move around, fight monsters (with a few minor glitches), and even leave the town. There are really only a few main areas left to implement, which includes sound support, logic for all the various spells, savegames, and the intro/ending cut-scenes. I probably won't return to working on it until after Rose Tattoo is finished, though. But when I do, I don't anticipate it will take long to finish the remaining areas, and then it would simply be a matter of playing the game through in earnest, identifying and fixing minor bugs as they're identified.
Of course, as an RPG, World of Xeen is a bit outside the scope of ScummVM proper. At that point, it may be time to finally launch the RPG sister project Strangerke and I have been wanting to do. :)
by Dreammaster (firstname.lastname@example.org) at May 22, 2015 12:09 PM
March 30, 2015
Rex Nebular, God's gift to the galaxy, has been stranded on a planet where he was attempting to locate to retrieve a valuable vase. To escape, he'll have to use the infamous Gender Bender to get in touch with his female side and successfully locate both the vase and a way to escape the planet.
The ScummVM Team is proud to announce support for Rex Nebular and the Cosmic Gender Bender, the first MADS game written by Microprose. It is now playable in ScummVM using the latest daily builds and ready for testing. As usual, all bugs should be reported to our bug tracker following our bug submission guidelines. While you play through the game, we would also love it if you could take some screenshots for us.
So, dust off your copy and try it out. If you don't already own the game, it can be purchased online at GOG.com.
by Paul Gilbert (email@example.com) at March 30, 2015 12:00 AM
January 25, 2015
December 31, 2014
We are proud to announce the release of a new version of ResidualVM, bringing
Myst III: Exile support, more than thirteen years after its original release.
Initial work towards adding Myst III support began back in 2009 as a side project.
A work in progress version of the engine was included in ResidualVM
two years later. The game became completable in 2012, but only now do we feel confident
enough to release a stable version providing a feature complete experience.
Additionally, some bugs have been fixed in Grim Fandango, and game data verification
has been added on first launch. So that you'll know if your game data was
copied correctly from your CDs.
Whether you are using Windows, Linux or OS X, you can check out our
downloads page, get yourself a copy of
ResidualVM 0.2.0, and decide later if you'd rather visit the land of the dead or
the land of a revengeful trapped man. The changes in version 0.2.0 are detailed in our
Instructions to setup the games can be found on our wiki.
As usual, even though we have tested the games thoroughly you might encounter game-breaking
bugs. So, please save often and report bugs to our
Our thanks go to Presto Studios, for releasing such a great game, and to the people involved
with testing ResidualVM over the years, allowing us to provide high quality releases.
2015 already looks like a promising year for ResidualVM, thanks to the ongoing effort
for adding Escape from Monkey Island support.
by bgK (firstname.lastname@example.org) at December 31, 2014 12:00 AM
December 23, 2014
We are planning to release ResidualVM 0.2.0 with support for Myst III as our second supported game.
The Grim engine has had quite a few changes since 0.1.1, so to avoid any big showstoppers for Grim Fandango, we need your help.
We need people to play through Grim Fandango and Myst III, preferably testing all the optional parts
that aren't required for completion too. If you are interested,
download our 0.2.0 pre-release builds (not the unstable builds),
and start playing.
Any bugs should be reported to our issue-tracker
on GitHub, and when you complete, you should post in our forums.
Details can be found here.
by aquadran (email@example.com) at December 23, 2014 12:00 AM
November 30, 2014
Yes, another Broken Sword post. In previous posts I wrote about my initial work to add support for the mac version of the Broken Sword game in ScummVM
and some more work I did to fix graphical glitches with this version. At that point the game was working fine for me. But soon, we got a report on the forum
that the speech was not working. Obviously it was working for me, I would have noticed if it wasn't. So what the heck?!
Bug reports are good. They stop me getting bored. And they show that other users have the mac version of Broken Sword and benefit from my work, which is also gratifying. So let's look at that issue. And to do so, first let's rewind to my first post. I wrote that I assumed the files with the same name (including the extension) as the files of the Windows version where in the same format, and in particular used the same endianness. And the files with a different extension were big endian in the mac version and little endian in the Windows version. That proved mostly correct (a few resources had been left as little endian data in files otherwise converted to big endian).
Except that wasn't correct. So why did it work? Because I had been lucky. When I initially worked on supporting that game it looked like my guess was correct, and I therefore made quick progress as I was not distracted by some strange behaviour. But...
Let's start the story from the beginning:
In the Windows version the speech is stored in a file named speech.clu
. There are actually two such files, one on each CD, and they store the speech as 16 bits compressed mono wave data. And as you can expect the data is little endian.
In the Mac version, the files have the same name (speech.clu
). So in my initial implementation I assumed the speech data was little endian in the mac version as well. And it worked... with the version I have (the French version). Obviously it didn't work with the version of the user reporting the bug (the English version) since the user reported hearing static noise instead of speech.
The two files (Windows and Mac versions, both English) have the same size:
But opening them in an hex editor shows differences:
Do the differences remind you of something?
If not go back and read again the first two posts in this Broken Sword mac support series.
Before looking at the differences, I will give a short explanation of the speech file format.
The speech files are a collection of sound resources. Each resource contains the wave data for spoken sentence and is organised as follow:
4 bytes: 'data
' (i.e. hexadecimal values 64 61 74 61)
4 bytes: number of samples
n bytes: wave data (16 bits mono)
So quite simple, but maybe no as simple as you might think. If you are thinking that n is the number of samples multiplied by 2 (since each sample takes 2 bytes) you are wrong. Because the data is compressed. This is not really important for now so I will keep the description of the compression for later.
What is important here is that we can see that the first 4 bytes after 'data
' are identical (in the image above hexadecimal values 8E E6 01 00 - which, since we know it is little endian, means 0x0001E68E = 124558 samples). But the values that follow are obviously a series of 2 bytes values for which the bytes have been swapped.
At this points, here is a small reminder in case you are not following me and did not go back to my previous posts: big endian and little endian are conventions used to interpret the bytes making up a data word (more at http://en.wikipedia.org/wiki/Endianness
). For example 42 in hexa is 2A, or when using two bytes 002A. When using the big endian convention, this would be stored as 00 2A. But when using the little endian convention this would be stored as 2A 00.
So this should be obvious to you now that both the Windows and the Mac version store the number of samples of the sound resource as little endian values, but in the mac version the data samples themselves are stored using big endian convention. Why mix endianness in the same file? Why do this for the mac English version but not the mac French version? Don't ask me, I have no idea.
Since some mac version use little endian and others use big endian, we need to know which one it is. Does it depend on the language, i.e. all French versions use little endian and all English versions use big endian? Maybe. But I don't trust statistics on a set of two samples. And what of the German versions?
Therefore we decided to use a heuristic to find out if the mac version the player has uses big endian data or little endian data. The heuristic works by computing the average difference between two consecutive samples (using absolute values). Using the assumption that a sound wave is smoother than picking values at random, the lower average difference is considered to be the correct endianness.
If we take the 13 samples from the example image above, assuming big endian for the mac version gives us the following curve:
The average difference value from one sample to the next is 425.17.
If we assume little endian data the curve is:
And the average difference value is 9344.75.
So in this case the heuristic tells us the data is big endian, which it is. Of course in the actual source code we use more than 13 samples to get a statistically valid heuristic value.
The original patch that adds the heuristic code can be found in the patch tracker: https://sourceforge.net/p/scummvm/patches/956/
But the story does not ends here. A new bug report
very similar to the original one (i.e. speech sounds like static noise) was reported a few months ago. It was quite obvious that the heuristic did not work for that user and the wrong endianness was used. Why is that? It turns out there were several issues with the original heuristic code.
And that is where explaining how the speech data compression works will help to understand what was wrong. The data for one sound resource is broken in blocks, each one starting with a number of samples followed by the sample values. When you have consecutive samples with the same value it uses a negative size and the value is stored only once.
So for example the following sequence:
0 0 0 0 0 1 2 3 4 5
would be stored as (where the brackets indicate the blocks):
-5 0] [5 1 2 3 4 5]
All those numbers (number of samples and sample value) are stored on 2 bytes.
Here is the original source code (if you don't see the source code visit the blog as it may not be visible in RSS feeds).
I will not show the uncompressSpeech() code (yet). The only thing you need to know is that it uses the value of _bigEndianSpeech as either big endian or little endian data. So what the code above does is get the data assuming little endian data and then compute the heuristic value for the samples it gets and for the same samples to which a byte swap is applied, which should be the value we would get assuming big endian data. Right?
Wrong! This heuristic forgets something: the data is compressed, and when uncompressing it always assume little endian when reading the number of samples for each block. But if the data are big endian this number would be different and the blocks would have different sizes, and because the block boundaries would be wrong it would cause number of samples to be interpreted as sound samples and some sound samples to be interpreted as number of samples.
For example let's look at the resource of 10 sample with the compressed data -5 0 5 1 2 3 4 5.
Assuming the data is stored in big endian, in hexadecimal values with two bytes per value this give us: 80 05 00 00 00 05 00 01 00 02 00 03 00 04 00 05
If we read this with the heuristic above, because the number of sample is always read assuming little endian data we get 80 05 = 1408 samples instead of -5 for the first block. So the code will get the following samples: 0 5 1 2 3 4 5 followed by 4 garbage values read beyond the end of the resource instead of getting 0 0 0 0 0 1 2 3 4 5. So if the data are stored using big endian, the heuristic values gets biased. Almost always it still gets a lower score as reading it as little endian though.
The solution here is to call uncompressSpeech() twice, once assuming little endian and once assuming big endian.
But there is still an issue with this heuristic.
When reading with the wrong endianness, since we may read the wrong length, it may for example be a big negative number. Because we are using a relatively small finite number of samples, statistically we could end up with a small heuristic value because it has a lot of consecutive samples with the same value. For this reason I made an additional change to skip consecutive samples with the same value when computing the average difference.
After that commit the value for the heuristic with the wrong endianness is consistently about 21000, i.e. 1/3rd of 16 bits integer range (65 535 / 3 = 21845). As noted by wjp: the average absolute difference between two random numbers drawn independently from a uniform distribution between 0 and N is indeed N/3. So this is quite reassuring.
So everything was correct after this change? No, that would be too easy. We want complex puzzles spanning several rooms, not some kind of hidden objects game. And the user reporting the bugs confirmed the bug was still present after that change. So let's look at a different room, or rather a different function.
Here is the code from uncompressSpeech():
Note something relevant? No? Look closer. You see it now? Yes, this function always give us the sound data in little endian format, whatever the endianness in which it is stored, and more importantly whatever the endianness of the computer on which the code is run.
And if you look at the heuristic code above, it assumes it gets data in the native endianness (i.e. the endianness of the computer on which the code is run). So when running on a computer using big endian convention the heuristic was wrong. Let's look again at our previous example and how it was interpret on a big endian computer:
Read with the correct endianess: 0 1 2 3 4 5 (duplicate values have been removed)
Was interpreted as: 0 256 512 768 1024 1280
Read with the incorrect endianess: 0 1280 256 512 768 1024 1280
Was interpreted as 0 5 1 2 3 4
So in the heuristic code, on a big endian computer we need to swap the bytes of the two set of data to get the correct value. This brings us to the final code, in which the heuristic computation was also moved to a separate function to avoid code duplication (since it is done twice):
The user reporting the bug confirmed he was using ScummVM on a big endian computer (a G4 mac) and that the speech was correct after that final change.
by Thierry Crozat (firstname.lastname@example.org) at November 30, 2014 09:38 PM
November 29, 2014
November 23, 2014
September 28, 2014
In my very first post on this blog I wrote how I came to be involved with the ScummVM
project by adding support to the mac version of Broken Sword 1. I wrote I had been lucky, and you will have to wait a bit longer to know why (yes I know, I am milking this one, but I promise I will explain it soon). I expertly avoided however to reveal that I had also been lazy. When I submitted the initial patch I knew the support was not perfect. I already mentioned it lacked support for AIFF music (and I will take this opportunity to correct myself: apparently the support was added by eriktorbjorn, at least according to the history on github
, and not by sev as I mistakenly wrote in my first post). But more importantly there were graphical glitches. Yes! GRAPHICAL GLITCHES! Oh, the horror! And I can't even claim I had not noticed them. That would mean admitting I was blind (or at least color blind).
The first one is visible every time you visit Nico in her apartment, which is quite often (just a shame you can't use that big bed). Notice anything wrong (no, not the bed)?
|George, don't leave! Have you seen what is waiting for you out there? A corridor painter in red! Stuff of nightmare! And the psychopath who painted that might still be lurking in a corner!|
Just in case the image above appears normal to you, here is what it should have looked like:
|No light in the corridor? I guess they forgot to pay the electricity bill.|
The second glitch is even bigger, although maybe not as obvious. I had not played the game for a few years myself when I added support for it in ScummVM, and while something was bugging me during my tests I was not sure what it was initially.
|Bull's Head Hill, Syria, on a murky day. The sky, the color of a swamp, was empty of any birds. And I was about to jump into the void.|
And here is what is should have looked like:
|A Sunny day in Syria. Maybe I will live after all. Not that it will stop me jumping though.|
So what is wrong? This scene in Syria has a background parallax layer, on top of which the foreground is drawn, with transparency where we should see the background. And you have probably noticed by now that the background was not visible in the mac version.
The game sometimes uses parallax layers to give a sense of depth to the scene. When the characters move on screen, the foreground and background will move at different speed. See wikipedia
if you have never heard of a parallax before.
This is the only scene in the game that has a background parallax layer. And as such it has a special logic for the draw code. Other scenes may have a foreground parallax layer however, as is visible in the video below.
The two glitches are caused by two different bugs. But they are somewhat related. The game uses 256 colors with a different palette for each scene. That means each scene defines a list of 256 colors, and then the image data is defined using the indexes (stored on 1 byte) in that list instead of using directly the colors.
The palette for each scene is actually defined in two separate resources: one that defines the palette for the scene itself and contains 184 colors (indexes 0 to 183), and one for the sprites that contains 72 colors (indexes 184 to 255). The first color (at index 0) is actually reserved for the top bar (inventory) and bottom bar (dialog options) area when the bars are hidden. It is forced to black in all the scenes, whatever the color defined in the data file. This is also the color used for the door in Nico's room. And it is used for the transparent part of the foreground image in the bull's head hill scene. And this is the index used for the transparency in the sprite data as well.
You have probably guessed it by now: the mac version does not use color index 0 for the door in Nico's room and for the transparency in the bull's head hill scene. After a bit of debugging it turned out it is actually using color index 255 (i.e. the last color of the palette instead of the first one). In Nico's apartment that color happens to be red, and in the Bull's Head Hill scene it happens to be some sort of brownish dark green. Once I knew what the problem was, it was fixed with a simple patch
Other than that the Mac version is identical to the Windows version. It still uses the first 184 colors of the palette for the background and the last 72 colors for the sprites. And it still uses color index 0 for the top and bottom bars area and the transparency in the sprite data. So I have no idea why they made that change for the two cases described above.
Here is the code to get the palette when loading a new room. As explained above it is called twice, once for the first 184 colors and a second time for the remaining 72 colors. We force color 0 to be black. Lines 6 to 9 corresponds to the fix for the mac version, in which we also force color 255 to be black.
And here is the beginning of the draw code. As I wrote above the Bull's Head Hill, which is screen 54, has a special handling. We first draw the background parallax and then draw the screen on top, skipping pixels with color 0 (which here means transparent). On line 21 we have the fix for the mac version, for which we also skip pixels with color 255.
And that is all for today. In the next post I will speak of the speech data, and I will explain why I was lucky in my initial implementation.
by Thierry Crozat (email@example.com) at September 28, 2014 06:22 PM
September 19, 2014
Hey everybody! :)
Finally, Sfinx got the point (with the English translation in parallel), that it's available for testing! :)
You can read the official announcement here
So if you have the time and you are eager to try something new, feel free to test our new engine!
by uruk (firstname.lastname@example.org) at September 19, 2014 06:54 AM
September 18, 2014
In this post I will continue to write about translating games for the ScummVM project. This is the last part of a three parts series.
Part 3: Translate a game into a new language
Some of the games for which we released a freeware version are from eastern Europe and were not released in English. So to give them a wider audience we decided to add an English translation.
The first such game was Dragon History
, a Czech game for which a GSoC
student added support in ScummVM in 2009, with the help of the original developer. The game was only released in Czech and Polish originally, but German and English translations have been added. If you want to know more about this game, see the official web site: http://www.ucw.cz/draci-historie/index-en.html
Since I don't know much about Dragon History myself, in this post I will focus on two Polish games from LK Avalon
. The first one Soltys
, is supported since ScummVM 1.5. It is available to download for free on our web site
, and in addition to the original Polish version, we have an English and Spanish translation.
The second game I will write about is Sfinx
. It is very similar to Soltys in the way it works, and support for it in ScummVM was added during this year GSoC
. We are currently working on the English translation and very soon (maybe tomorrow?) we intend to make it available so that non-Polish ScummVM users can test the game, report bugs and also suggest improvement to the translation.
Edit: the call for tests is now live
Both Soltys and Sfinx have two data files named vol.dat and vol.cat. The latter is a catalog that lists the files present in the former and at which offset they start. So when the game needs a file, it can look into the catalog where to start reading it in the vol.dat file. To edit the data files however, we need to extract those. Then we can repackage them into a new vol.dat file, generating a new catalog file as well in the process. We have two tools
to perform the extraction and packaging, and they work for both Soltys and Sfinx (despite some minor differences in the file format).
Once uncompressed, you will have a lot of files. All the dialogs are in a file named CGE.SAY. The hotspots names are in the files with the SPR extension. The other files can be ignored (they will be needed when repackaging the game though.
So what does the CGE.SAY look like? Here is a small portion of it that shows almost everything there is to know:;--Anna above. 1:22=Oh, what a nice pussy!|I would love to have one;--Vincent in the dark 1:31=Where's the light? I can't see 1:32=There should be a shutter,|let's try to lift it
;--Vincent about the cleaning stuff 2:01=Cleaning? Never!|It's for the girls!;--Anna about the cleaning stuff 2:02=Isn't there a gentleman around?
Lines starting with a semi column are comments. There are a lot of them, which is a great help.
Dialog lines start with xx:yy as you can see above. The xx is the room number. So in the example above we have a portion of the dialogs for the first two rooms. The yy is the text number in this room.
The pipe indicate a line break. So for example the first text of the second room will look like this in game:
Simple, isn't it?
Now let's have a look at one of the SPR files, for example 02ZSYP.SPR. As the name suggest this is one of the hotspots in the second room. The start of the file look like this in the polish version:Type=AUTOName=zsyp na <98>mieci
[seq] 0 -2 0 0 0 8 1 3 84 2 127 8 .OTWIERA 1 0 85 2 127 8 .ZAMYKA
2 -2 0 0 0 8
[ftake]say -2 2:5 brudny
[mtake]reach -2 2:7 . zsypSOUND 2:7 2:84pause -1 72SAY -2 2:4NEXT -1 0 . smiec popycha
The name is what appears on screen when moving the cursor to the hotspot. We can now also see that the file is named after the hotspot name. This makes it easy to find a file when you know the hotspot name... in Polish (not so easy when you know it in English ;) ).
The <98> is the way my text editor displays non ASCII characters using their hexadecimal value (so in decimal we have here character 152). In this case the character is ś. The game is using the CP852 encoding
(with only the example above it could also have been using the mazovia encoding
, but other characters allow to make the distinction). Fortunately English does not use many non ASCII characters, so we don't have to deal with this much.
So, the polish name is zsyp na śmieci
. Google translate
tells me (I don't speak Polish myself) that it translates into garbage chute
. So let's modify the second line in the file and see how it looks:Type=AUTOName=garbage chute
For Sfinx, the bulk of the work was done by Strangerke and then I made a couple of passes to improve the English and fix spelling mistakes. Uruk, the GSoC student who worked on the engine, also made some modifications.
For Soltys, the Polish to English translation was done by neutron and the Spanish version is from IlDucci and The FireRed. I am currently working on a French translation as well.
The process I explained above is therefore very similar to what I explained in the previous post to improve an existing translation for Drascula:
- Unpack the data file.
- Edit the dialogs and hotspot names.
However there is one major difference. Because the game was only released in Polish in the first place, the font data does not contain all the characters we need for other languages. For English this is not an issue, unless you happen to use a word loaned from French, such as déjà vu or café. When translating to French however you need those accentuated characters. So there is one more step to do: modify the font data (which was done by Strangerke on Soltys).
The font is stored in a file called CGE.CFT. This is a simple bitmap font, for which each pixel is black (or another color) or transparent. So we need one bit to store a pixel. If the bit is 1, the pixel is visible, and if the bit is 0, the pixel is not visible. The height of the font is 8 pixels, which conveniently can therefore be stored on one byte (because in case you don't already know, 1 byte contains 8 bits). The width is variable, and if for example a character is 4 pixels wide, thus 4x8 pixels, its data is coded on 4 bytes. And there are 256 possible characters.
The font file starts with the width, coded on one byte, for each characters. That takes the first 256 bytes. Then the bitmap starts. Here is the start of the file for Sfinx displayed with hexadecimal values. The first column is the address (also in hexadecimal). We have 16 bytes on each line. A star denotes one or more lines that are identical to the previous line.0000000 04 06 06 06 06 06 06 04 04 04 04 04 04 04 04 040000010 04 04 04 04 04 04 04 04 04 04 04 04 04 04 04 040000020 04 02 04 06 04 05 05 02 04 04 03 04 02 03 02 030000030 04 04 04 04 04 04 04 04 04 04 02 02 04 04 04 050000040 05 05 05 05 05 05 05 05 05 02 04 05 04 06 05 050000050 05 06 05 05 06 05 04 06 04 06 05 03 03 03 04 050000060 04 05 04 04 04 05 03 04 04 02 03 04 03 06 04 040000070 04 04 04 05 03 04 04 06 04 04 04 04 02 04 06 060000080 04 04 04 04 04 04 04 04 04 04 04 04 04 05 04 050000090 04 04 04 04 04 04 04 05 05 04 04 04 04 04 04 0400000a0 04 04 04 04 05 05 04 04 05 05 04 04 04 04 04 0400000b0 04 04 04 04 04 04 04 04 04 04 04 04 04 05 04 0400000c0 04 04 04 04 04 04 04 04 04 04 04 04 04 04 04 04*00000e0 05 04 04 05 04 04 04 04 04 04 04 04 04 04 04 0400000f0 04 04 04 04 04 04 04 04 04 04 04 04 04 04 03 040000100 00 00 00 00 1e 29 2f 29 1e 00 1e 2b 2f 2b 1e 000000110 0e 1f 3e 1f 0e 00 0c 1e 3f 1e 0c 00 1c 5b 7f 5b0000120 1c 00 1c 5e 7f 5e 1c 00 ff ff ff 00 ff ff ff 000000130 ff ff ff 00 ff ff ff 00 ff ff ff 00 ff ff ff 00*0000180 ff ff ff 00 ff ff ff 00 ff ff ff 00 00 00 00 000000190 2f 00 03 00 03 00 14 7f 14 7f 14 00 26 7f 32 0000001a0 13 0b 34 32 00 1a 25 1a 28 00 03 00 3c 42 81 0000001b0 81 42 3c 00 06 06 00 08 1c 08 00 60 00 08 08 0000001c0 20 00 38 07 00 3f 21 3f 00 22 3f 20 00 3b 29 2f00001d0 00 31 25 3f 00 0f 08 3f 00 37 25 3d 00 3f 25 3d00001e0 00 01 3d 03 00 3f 25 3f 00 37 25 3f 00 24 00 6400001f0 00 08 14 22 00 14 14 14 00 22 14 08 00 02 29 050000200 02 00 1e 21 2d 0e 00 3c 0a 09 3f 00 3f 25 26 180000210 00 1f 21 21 12 00 3f 21 22 3c 00 3f 25 25 20 000000220 3f 05 05 01 00 1e 21 29 19 00 3f 04 04 3f 00 3f
If we look at the first few lines, we can see that the characters are between 2 and 6 pixels wide.
Let's try to have a look at the start of the alphabet. In the ASCII table
, we can see the value of the letter A is 65, and since values start at 0, that means this is the 66th character. So first we will compute the sum of the widths of the first 65 letters.
That would be 4 + 6 + 6 + 6 + ... + 4 + 4 + 4 + 5 = 263
So if we skip the first 256 bytes (the character widths) and then the next 263 bytes, we should get the data for letter A. So let's look at the data that starts at address 256 + 263 = 519 (207 in hexadecimal).
I have highlighted in red above the width for the 66th characters, which as we can see is 5, and the 5 bytes starting at address 0x207.
Let's write them, with the corresponding binary representation below (with the least significant bit at the top): 3c 0a 09 3f 00 0 0 1 1 0 0 1 0 1 0 1 0 0 1 0 1 1 1 1 0 1 0 0 1 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0
So now a bit of ASCII art: we replace the 1 by a @ and the 0 by a space @ @ @ @ @ @ @ @ @ @ @ @ @ @
You recognize something?
Just for fun, let's do the same for the next two letters: @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @
So we can edit the font file using an hexadecimal editor for example. This involves some ASCII art (exciting :-), and it can be challenging to fit an accentuated characters on 5x8 pixels), some additions on hexadecimal numbers and some conversions between binary and hexadecimal (boring :-( ).
This concludes my three parts posts on translating games for ScummVM. I hope you found it interesting. Now I will take some rest while you start testing Sfinx. There is one last thing though: ScummVM is a community effort, and it does not only involves software developments. You can contribute in other ways, such as translating freeware games, translating ScummVM itself or helping with the user manual
. So if you are motivated to help us, please get in touch for example on our IRC channel (#scummvm on irc.freenode.net) or forum
by Thierry Crozat (email@example.com) at September 18, 2014 09:23 PM
September 16, 2014
In this post I will continue to write about translating games for the ScummVM project. This is the second part of a three parts series.
Note: This post contains an embedded sourc code example that is not visible on the RSS feed.
Part 2: Improve the original translation of a game
Sometime the official translation of a game could be mistaken with the result you would get from the AltaVista translation of the 90s
. And I am not exaggerating.
The French version of Drascula
was an example of this, and I am told the Italian version was not better - but my limited knowledge of the Italian language does not allow me to confirm. While this was hilarious in its own way, it distracted from the game, so we decided to provide an improved translation for both Italian and French. I didn't work at all on the Italian translation, so the examples I will take are all from the French translation. But most of the explanation would work for the other translations.
Here the strings are partly in the game data file and partly in the original executable. We extracted the string from the original executable and instead in ScummVM they are in the drascula.dat file that we provide with ScummVM. So improving the translation meant both modifying this drascula.dat file and modifying the the game data files. Sometimes it also meant adding new strings, as for example the subtitles for some languages were missing in the Von Braun cutscenes
Modifying the strings in the drascula.dat file is easy. The strings are hardcoded in the source code of the tool used to generate that file
. So we just need to modify that source code. The only little difficulty is that non-ASCII characters (e.g. accentuated character, and we have a lot of those in French) are using the Code Page 850 encoding
. And in C we need to use the octal number in the string preceded with a backslash. So for example, to have an è, the decimal value in the CP850 encoding is 138, which in octal is 212. So the string would be '\212'. Therefore to get "Chèvre" ("Goat" in English, I think my brain was permanently damaged by working on Broken Sword) I would need to write "Ch\212vre".
Modifying the data file is not much harder. Those files are actually ARJ archives
. So you can easily decompress them using a tool that supports this compression. Files with strings are those with the extension CAL (which contain the dialogs) and ALD (which contain the hotspots). But you cannot edit them directly; that would be too simple. They use a simple encryption: each byte is x'ored with 0XFF
For example the letter A in ASCII has a value of 65. In binary this gives 01000001.When you x'or it with 0XFF (11111111 in binary) this gives: 10111110 (190 in decimal).
To get back to the original text you just need to X'or it again by 0xFF.
So I quickly wrote a simple C program to decrypt and re-encrypt the files:
To give you an idea of how bad it was, here are some of the hotspots from the original version and the corresponding ones from my improved version.
|PUIT||PUITS||A simple typo you might think. Maybe, if it had been the only one...|
|CIMTEIERE||CIMETIERE||Anagrams now? Maybe that was actually designed as a puzzle?|
|CAISSON||TIROIR||Where did that come from??? Canadian French maybe?|
|CERVEAUS||CERVEAUX||You may need a brain to know that the plural of words ending in 'eau' takes a X and not an S.|
|TRONC||COFFRE||Maybe my favorite. It make me think that the "translator" may have been working from the English text and not the Spanish one. TRONC is a tree trunk. COFFRE is a chest... or a car trunk.|
|ARMARIO||ARMOIRE||OK, they forgot to translate that one.|
|BAUL||COFFRE||And that one.|
|ESPEJO||MIROIR||And also that one.|
|PUERTA||PORTE||Did I download the Spanish version by mistake?|
And you have many more like this. And the dialogs were not much better. For those who understand french here are a few examples of original dialogs:
- Quelle merde de jeu dans lequel le personnage principal meurt! Un instant, qu'y a-t-il de mon dernier désir?
- Et bien merci et au revoir. Que tu la dormes bien.
- Non rien. Je m'es allais déjà.
- Comment peux-je tuer un vampire?
- Qu'est-ce qu'on suppose que tu fais?
I will stop there. But I could fill pages like that. So if you speak french and fancy a good laugh, feel free to download the original french version (not the updated one) from our web site
and play the game.
Another game for which we improved an existing translation is Mortville Manor
. This is a French game that was also released in German and English. Except the DOS version was never released in English. Strangerke (one of the developer who worked on the engine in ScummVM) extracted the English strings from the Amiga and Atari version. But it was still missing all the dialogs. Strangerke created a Google Doc spreadsheet with the French and English strings and with a ScummVM user named Hugo we started fixing the existing English translation and translating the missing strings. Then we implemented a small tool
to generate a data file from these strings (mort.dat, which is distributed with ScummVM) so that users can play in English using the game data files from the DOS French or German version.
For Mortville Manor, we actually also bundled the French and German strings and the data for the menu in the mort.dat data file. That way we can easily improve those languages as well. But for now they have not been improved and only the original French and German versions are available. I have been told the German one is not perfect. So if you like this game, speak German, would like to improve the German translation, and have a lot of free time on your hands you can contact me ;-)
See you tomorrow for part 3.
by Thierry Crozat (firstname.lastname@example.org) at September 16, 2014 07:23 PM
September 03, 2014
Adding support for Myst III: Exile to ResidualVM is nearly done.
Now, we only need your help to find the last rough edges by playing the game thoroughly and reporting any issues you encounter.
If you are interested, locate your copy of the game, and download a daily build (not the 0.1.1 release build).
Instructions to setup the game can be found in our README.
Bugs should be reported to our issue tracker on GitHub.
Once you have finished, please report your testing results in our forums.
by bgk (email@example.com) at September 03, 2014 12:00 AM