Planet ScummVM

Hi, it has been a long time since my last update.
TeenAgent has been implemented into the main trunk of ScummVM, however it is not based upon my engine, instead Vladimir approached the ScummVM team a few months back with a rather complete engine, the engine itself is very tidy and the game is in a completable state.
Congratulations are in order to Vladimir for his incredible work on the game, surpassing any progress I would have been able to make with my poor reversing and coding skills.
The project was great fun and an incredible learning experience while working on it. I would like to thank _sev, Buddha, salty_horse and jvprat for their patience with me while I was learning.
I would also like to thank john_doe for his commits for the music support, as TeenAgent used a strange and new module format for music. I have great respect for everyone who helped me learn. It will not be my last attempt.

I have recently been looking into the code used for Maniac Mansion sound effects on the Apple II.

A little history on the Apple II:
When Maniac Mansion was ported to the Apple II(around the same time as the C64 port) it was seen as a very ugly version, in comparison to the C64 version. It had some graphical issues and didnt include scrolling. Unfortunately the port also suffered in a worse way. While the C64 had the SID chip, an incredible sound chip allowing for some really interesting sound effects, the Apple II port had to make do with a REALLY basic speaker. The Apple II port used the speaker to generate some sound effects such as the character selection 'click'. Unfortunately the speaker could only output a square wave 'click' when the port was addressed(using a 'bit $C030 command) The speaker itself had not options for changing the output or frequency of the output.
This of course left games programmers with very little to play with, many of them came up with intelligent code that would call the speaker multiple times using various loops to 'click' their way into something vaguely audible. Maniac Mansion was no different. Lucasarts created a resource format which consisted of a collection of various bytes which told the code how it should 'click' the speaker.

The resource format is incredibly simple and the resource is simply read in by the code which judges how often and when it should click the speaker.

I am currently about half way through reversing the code(6502 ASM), I have converted about half of it to c++ and am planning on finishing the rest within the next few months(there isnt much, I just do not have much time with university commitments, as it is my final year any external work has to sit on the backburner until I get stuck with my university work :p)

I would like to thank Tobias for his patience with me, he has been especially helpful as he has not ever given me the answer, rather he provides ways for me to find the answer and makes sure I understand it. He has been so helpful in explaining some of the 6502 opcodes and explaining how they were used. I am greatful for his friendship and advice.

Though this code is not very complicated, I am still struggling to understand completely what it does. I am sure this will come to me.

EDIT:
I have documented all I know about the resource format on the ScummVM wiki here : http://wiki.scummvm.org/index.php/SCUMM/V0/Maniac_Mansion_Apple_II

Though this information is pretty irrelevant, since the code is what actually makes sense. The resource is simply a loop controller.

Finally, the support of Fascination is showing some interesting progress! These last two weeks, the window behavior has been fixed, the hotspots correctly handled in them, and it’s now even possible to play through some levels. DrMcCoy even added a save/load function that works perfectly! There are still a lot of glitches in the floppy versions (almost none in the CD version) and it’s not completable for various reasons, depending on the platform and language concerned.

Here are some screenshots of the CD version.

Stay tuned for further news! I promise I’ll try to give some before this summer

Every once in a while, I check on eBay for a few games that I either find interesting, might help ScummVM in some way, or because I'm trying to track one down. I was looking for Mac versions of Sierra games, and at one point I happened to scroll down to the bottom of the screen and something caught my eye. "Leisure Suit Larry" (along with "Lego Batman") was mentioned on the bottom of the screen:


Bewildered as to what that link could possibly mean, I clicked on it and it lead me to here. Apparently Leisure Suit Larry is a popular item on eBay. Or something. I don't really get it. Not to mention none of the products listed below the description are even related to LSL.

</random blog post>

EDIT (2/3/10): My continued eBay searching has left me somewhat confused. Sometimes the item is just ridiculous, sometimes the price is just ridiculous. Since I don't want to just create another post, here are a few oddities I have found:

- King's Quest I CD: In addition to the high price... "replaced with the CD version"
- King's Quest V FM-Towns: High price. I would love to have it to be able to support it though! (Will accept a donation :P)
- Indiana Jones FM-Towns... Phone Card: Self-explanatory
- LucasArts Japanese Mac Pack: I would love to have this too (not as much as KQ5 FM-Towns), but I don't want to spend this kind of money...
- Indiana Jones and the Last Crusade: Seems normal... until you see what it comes with.

And, the icing on the cake:

- King's Quest VI "Girl in the Tower" CD: My jaw dropped when I saw this one. Yours probably will too. The price pretty much tells all. But it has free shipping!!!!


In other news, I've been tinkering with the Mohawk QuickTime code and working on adapting it to the Graphics::VideoDecoder interface in ScummVM. The more I do this, the more it feels like shoehorning the features that QTPlayer has into VideoDecoder isn't working. In addition, md5 mentioned that IHNM Mac uses QuickTime MIDI files. Curious as to what that involved, I looked up some samples on the internet. They use the good bad ol' QuickTime format to hold its own MIDI format (I had hoped for SMF in a QuickTime wrapper). Here's what Apple's QuickTime specs said about it. Anyone out there have better info than that?

Those of you who have been following the IRC channels will know that I made some real progress in the disassembly of the tSage engine, used by the Ringworld - Revenge of the Patriarch game, as well as several others. I hadn't really gotten into any of the game logic, but I'd made great strides in decoding the image formats and graphics display, so much so that my engine on Scummvm-misc could display test dialogs, as well as the right-click game dialog with proper highlighting and everything. Sample screenshots below:





However, given the perversity of life, I should have known better than to announce on the channel that I was making real progress. :(. Because as I was getting into the game execution logic, I began to realise just how complicated they made the engine. Whilst using C++ has it's advantages from the perspective of RE, since it does group related functions close together with the variables they use, in this case, they seemed to go overboard with it.

A preliminary look at the main event loop of the game shows they implemented a stack of "action objects" - multiple independent event handlers can be registered and control swapped form one to the other. I'm not sure, but I presume that this is to allow for things like cutscenes, where standard game operation is superseded.

For even more complexity, they went with a class hierarchy for the action objects which is at least four levels deep!.
The classes, as far as I've been able to determine, are as follows:

• They have a core 'SavedObject' class, which has some internal logic for registering each object instance in a central list. I presume this is for automating saving and loading.
• A child class which I'll call "SmartPointer". This class seems to encapsulate a pointer to a secondary object, and provides core logic for 'fixing' up the pointer reference. I can't be sure yet, but there's a lot of 'fixup' code in the engine, which I presume handles restoring pointer references between objects when a savegame is loaded, and all the objects are re-constructed. This class also implement a basic virtual table of processing and dispatch methods that automatically defer to the foreign object if one is set
• Yet another child class which introduces basic event manager functionality amongst other things
• A concrete fourth child class which seems to further override some of the event manager functionality, and is what gets added to the game's "action list".
  

As you can imagine, all this makes it somewhat different to understand what the blazes is going on. I've also examined the engine from some other points, such as scene loading, and even that seems to be overly complex, with method calling method - they really went in the direction of generalised/generic functionality, with the result that the code seems to be much more complex than it should otherwise need to be.

</whinge>

In any case, the tSage engine was only meant to be a part-time RE effort for when I needed a break from the MADS engine. I did somewhat get carried away with it for the last few weeks, but that was only because the areas I was looking at proved so easy to reverse engineer. Now that I'm running into more complex code, I've decided that the break's over, and I'm returning to work on the MADS engine again. For that, at least, the code seems somewhat simpler, even if it can be a pain to run in the DOSBOX debugger because the segments keep getting shifted around in memory. ;)

For those of you who are Ringworld fans, don't worry, I'll still be working on the engine. It's just there won't be as rapid a progress on it as I'd recently hoped.

Today I am once again writing on something about SourceForge.net that bothers me. Just to get this straight first: I use SF.net and a lot, and in general it works fine for me; but in the past year or so, more and more things keep piling up that really annoy me about it. And so I feel the need to vent some of my frustration by telling the world (or at least the handful people reading this blog ) .

Now before you yell at me “Don’t blog about it, dude, report it to them!”: Of course I report all issues I find via their tracking system or their IdeaTorrent. All right and proper. It just is that it recently most of the time feels that all the reporting is for nothing… I report things, and they get ignored; or they get lots of attention and in the end are still closed with “Won’t fix, this is as intended” or “We currently decided not to fix this”…  To give another example, just today, I went to their IdeaTorrent, and noticed some duplicate ideas. IdeaTorrent allows you to report duplicates. Sweet! Until you realize that lots of people report duplicates, but they never get resolved… I am not sure whether to cry or laugh when I read this on the page where you report duplicates: ”The admins will accept or reject the merge shortly.” (Emphasis mine). “Shortly”, right…

Anyway, back to the subject: Some months ago, SF.net added a “project review system”. It allows anybody to give any project a thumbs up or down, and optionally a review. At first my thought was, neat idea, this could help filter out all the junk, and let the few real gems shine.

That is, until I realized that anybody can vote, even people not logged in. So you can quickly abuse it to generate many thumbs. Moreover there is nothing a project admin can do to delete inflamatory, off-topic or simple insulting “reviews”. Don’t get me wrong, of course negative comments must be allowed. If you try a product and it doesn’t work well for you, you should of course be allowed to give a negative review. But a constructive one, please! Take libpng for example: At the time I am writing this, they had 26 “thumbs up” and 21 “thumbs down”. With “reviews” like “YOU SUCK!!!!!!!!!!” or “that sucks man”. I kind of doubt that the reviewer, “anonymous” (who would have guessed…) is basing this on actual user experience. Well, or if he does, after this “review” I sadly don’t know what the experience actually was. To me it looks more as if it has beome a sport for various people to thumb down random projects. Man, I am glad that ScummVM (444 thumbs up, 92 down) and Fink (37 up, 6 down) are not hit as badly. Although I really wish I knew what those 92 thumbs down guys were complaining about. Maybe some of them even left reviews explaining their gripes, but so far, I haven’t found a way to read any reviews except for the last 25, so I’ll probably never know…

To sum it up, I just wish this project review system wasn’t so horribly broken and… naive? unprofessional? incomplete? useless?

There are certainly many ways to improve the situation, and luckily not all of them involve removing this feature. Instead of writing about those in detail, I’d like to point you to the relevant IdeaTorrent page, and ask you to vote on the various proposed solutions there. Yeah, I started this post with a complaint on how ideas on this IdeaTorrent may not be read at all… but I haven’t lost hope yet, and one should at least try, right? . So, let’s all try together, maybe we can move this unmovable rock a little bit.

P.S.: In case you are wondering which IdeaTorrent idea I found duplicates for, it was of course the one I linked to above… Seems quite some people are annoyed by these reviews and post new ideas suggesting improvements. If you wonder why they keep posting new proposals instead of voting on or adding to existing ones, well, my guess would be that it’s because the IdeaTorrent’s “Search” button, quite surprisingly, doesn’t do what it says. My request to rename it to “Filter by tags” (that’s what it actually does… not quite the same, nor quite as useful, don’t you agree?) was, of course, rejected long ago .

To coincide with the updated ScummVM logo, ScummVM t-shirts have also received a touch up. The shirts sport the verb list from Maniac Mansion on the chest, the updated logo on the upper back, and the scummvm.org url on the left sleeve. Shirts are now available in sizes small through 3XL. If you need a larger size, please contact sales[at]combobreaker[dot]com with a request.

Shipping to the USA and Canada is free, and shipping elsewhere is offered at a discounted rate. Head over to Combobreaker.com to pick them up!

Hey, here’s another small video, this time with Mike walking in-game (without pathfinding) and talking to Hank in the diner. Also a nice showcase for most of the remaining graphical glitches (highlighted in the annotations) :

And so, 2009 comes to a close. For those of you who have a good memory, you might remember a post I made last year about what I had hoped to accomplish this year. Let's take a look:

2009 Goals: (In order of probable completion)
- Fix Cinepak decoder!
- Fix QuickTime IMA ADPCM!
- Get background videos working!
- Figure out image format for other Mohawk games (Current hunch: LZSS similar to Myst format)!
- Finish Riven support (aka all the hardcoded puzzles :P)!

I have crossed out what was accomplished. Technically, I have a WIP patch that allows for background videos to work in Riven and background videos already work in Myst. So, we might consider that one done, even on a bit of a stretch. :P And now for next year:

2010 Goals: (In order of probable completion)
- Finish Riven support (external commands, mostly)
- Finish Myst support (some resource fixes, and then lots of the hardcoded opcodes)
- A look into other games more

Again setting lofty goals! We may even consider them monolithic in size! (Pun intended!) And for one last treat, to see the "good" ending of Riven, open the debug console in ScummVM (ctrl+d) and use these commands:

var ttelevalve 1
var ttelepin 1
var ttelecover 1
var pcage 2
changeStack tspit 137

Turn the lever down and press the button and voila! (Note: The DVD version may have to use a different card number, but just walk over to the telescope if you end up elsewhere :P)

Happy New Year! :)

Today, December 29, 2009 at 23:23:31 (UTC), Mohawk entered the main ScummVM tree. You can now go grab a daily build and try out Myst or Riven. And now Strangerke can make his own gameplay videos and not coerce me into making them. :P

So, this is the end. Mohawk is now in the trunk and everyone can attempt to play Myst and Riven in ScummVM. :)










Or, perhaps the ending has not yet been written...

Instead of continuing working, I’ll rather goof and play around with video capturing . So you can now gaze at some working Mike walking animations:

Just a minor update: Thanks to sev, who pointed me to the right place in the wine sources, I managed to solve the cursor inconvenience .

The problem was, of course, my own damn fault; I misunderstood the nature of cursor resources. They’re not, as I assumed, plain cursors, one by one each taking up a resource entry. Instead, there are cursor groups, each capable of possessing a variable amount of cursor images. That isn’t even a quirk of NE files, it’s like that in PE files as we well, so my complaining about the executables being in that old format doesn’t apply either. Mea culpa.

Initial reports claim the gold held in the MEGA-BANK vault just began to vanish! MEGA-BANK Security were asked to review the CCTV footage, however they were unable to see anything.

Do you think you can solve this mystery? Are you willing to take the role of Mark Hopper? A teenager hired by the RGB to infiltrate the mansion of John Noty and discover why and how he is becoming so rich.

That's right ladies and gentlemen, TeenAgent is now ready for testing! Why not take this opportunity to join Good Old Games where you can download TeenAgent for free. You will also need a Subversion build of ScummVM. As usual any bugs should be reported to our bug tracker. For further information, check out the bug submission guidelines.

The usual direction when adding support for T7G and 11H to ScummVM is to take the data files, decode them, and get the video and/or audio out of them. Last week in a moment of boredom, curiosity, and 'how hard can it be?', I decided to work on working on going the opposite direction and have a look at encoding instead.

As it turns out, still frames were fairly straight-forward to implement due to the simple structure.

Animation frames were a little harder. Most of the issues resolved to be related to how I was attempting to keep track of palette changes between frames (and the rewriting of the new frame to line up colours common between both palettes but at differing indices), but I also got caught up in some minor logic errors in the encoder (in particular, encoding the same tile using two different methods, and requiring a "new line" opcode at the end of *every* line - not just those that end early). The final result can be found at this YouTube video (sorry - no embedding, since my blog doesn't support it).

Encode speeds currently aren't the greatest yet, the output VDX files are not compressed, and the encoder requires 8bpp Indexed BMPs for use as input, but it's certainly a start for proof-of-concept purposes :)

After about a year of work, videos finally work and now with full a/v sync (md5 suggested something that could be the problem and that was it). Of course it was about a five line fix, but that's the not the point. ;)

Mohawk progress in the past month, let's see... I don't really remember a lot of it. I've spent most of my time on Uni things. It's now break, so it's time to work on more! digitall has been working on QDesign Music 2 (a complex audio codec used in Myst ME) and then we've taken an early look into the Living Books games. Nothing playable yet, but we can parse the "book outline" files.

I'm sure there's more I've forgotten, but I'll give a quick wrap-up:

Myst: Playable
- Many graphical glitches
- Wrongly disabled hotspots
- Only Stoneship is accessible from Myst Island
- D'ni is accessible! Can't hand a page over though :P

Myst ME: Slightly less playable than Myst (inherits the same problems)
- Less videos work
- Probably some PICT problems that have yet to be encountered

Riven: Very playable
- Almost no status update since last time :P
- Fixed one minor regression from cleanup

Myst is movin' right along now! Unfortunately, both Myst and Riven have hardcoded puzzles which are the biggest impediment in progressing further. But fear not, they will be worked on!

The year is 1995. The place is the Czech Republic. You find yourself surrounded by inventive minds producing the latest Czech point-and-click adventure, and historically the first Czech CD-ROM-based adventure game. You are Bert, a young dragon with a taste for adventure. After being corrupted by an evil magic wand named Eveline, you are desperate to remedy the mischiefs you created. You decide to set out to find your father who disappeared a while ago.

Grab your copy of the Dragon History game and a Subversion build of ScummVM, and get playing! The game is available in Czech, English, and Polish so you have no excuses!

All bugs should be reported to our bug tracker. For further information, check out the bug submission guidelines.

The Scummvm-misc listing welcomes the new work in progress Dark Seed 2 engine by DrMcCoy. It looks very promising.

I've returned to spending a lot of my time working on the disassembly of Rex Nebular, and implementing it's functionality into the M4 engine. It's likely going to be an extended process, because the game engine is unbelievably complex. It likely comes from the MADS engine being designed as a generic game engine rather than just for single games specifically.

For example, I only just last night disassembled some of the code associated with displaying messages. Any given message is represented by a unique 'Message Id', which in turn loads a message resource that can contain an entire series of control elements for formatting dialogs. A sample message resource for looking at the curtains in the starting room is as follows:

[title25][sentence].Wincing at the "Drapes Of Wrath", you swear
to use them to light the fire when
you burn your ex-decorator at the stake

I'm dreading how complicated the actual script interpreter is going to be. :P

A new nice port is in the works:It is the very first screen shot. Hkz is working on it. ScummVM portability is demonstrated at its most.

Update Nov. 26: SourceForge.net finally fixed the voting issue. Yay! I still don’t understand why they didn’t manage to post a warning sign for the whole period of this, but for now I am just happy this is resolved. So now it’s easier for all of you to vote on the ideas I linked to below (and all others, too ).

SourceForge.net has been busy changing lots and lots of things in the past years. Some things certainly got a lot better by this. But some really got a lot worse. Well, when renovating a system, that happens; usually such “regressions” are simply fixed quickly, and one we go.

Not so with SourceForge, it seems . For example, they shelved from their old “request for enhancement” tracker (on which you could suggest your ideas on how SourceForge could improve their site, have it sit there for some years, then watch it being closed with a comment stating that they will consider it at some point. Yes, I am bitter :-/). Instead of the good old RFE tracker, they now use a new way to track feature requests: They switched to using IdeaTorrent, a fancy pancy Web 2.0 thingy which allows people to vote on issues! So, the community can put emphasis on what features are important to many people, and thus this helps SourceForge to prioritize things.

Sounds great, doesn’t it? Only, it doesn’t work. Voting on SourceForge’s IdeaTorrent has been silently broken for over two months now. Silently meaning that it looks as if your vote has been registered, it increases the count, no error. Only when you reload, you notice your vote was, in fact, ignored. This has been reported via their Trac and via their IdeaTorrent in September. And they haven’t even managed to put up a simple box on the IdeaTorrent page that warns people about this, and points them to the workaround. (I requested this highly “innovative” workaround September 21).

In principle, SF.net staff has been great in personal communication on this issue; they gave me a “VIP” treatment and even skyped with me about this. Only, in the end, the result is still the same: For two months, people would vote on the SF.net IdeaTorrent, and their votes were silently lost. And not even a warning sign (something that should take virtually no time, even with testing before deployment) has been put up yet.

Wow. I am baffled. Something must be really wrong with their internal organization, I have no other explanation. Lack of internal communication? Strange priorities? Incompetence? I am trying to think of an explanation that doesn’t let them look quite incompetent, but just can’t come up with any (if you have a better imagination than me, feel free to leave a comment).

I have been using SourceForge for many years. I used to be a big fan and supported. I used to defend them whenever people raved about how SF.net sucks. I still use it a lot, but I get more and more annoyed with how things are these days, and how slow things move. And how unimportant the wishes of the developer community seem to be (but maybe it’s just me who perceives it this way). Normally, I’d keep silent about this and hope for a fix, but I kind of have lost hope on that by now. A sad day for me .

Anyway, if you haven’t given up on them, consider voting on some of the following IdeaTorrent entries (but don’t forget to use the workaround, else your vote is sent to /dev/null):

• Fix the ideatorrent voting bug
• Add ReviewBoard to Hosted Apps
• Allow importing of data for hosted apps.
• Support ssh authentication for subversion
• Science/Engineering category should be on the home page! Instead of say, “Clustering”…
• Migration tool from SF.net bug tracker to MantisBT or trac
• Statistics should include hosted apps activity (trac at first)
• Thumbs down is not appropriate for free software
• Lets add Bugzilla to hosted apps
• Trac’s SSL urls are not friendly to newbies

And maybe I should recycle a rejected support request of my own: “Add real searching to IdeaTorrent”. Because the “search” box in IdeaTorrent apparently doesn’t actually perform a search. Instead it filters based on tags. Intuitive… But I was told that this is the “intended behavior”, and answer I get a lot these days. Maybe now you have a glimpse as to why I am unhappy .

I am a massive fan of point and click adventure games and although there was a spell where developers seemed to avoid the genre like the plague, there is certainly a noticeable return with titles such as Monkey Island, Broken Sword, Simon the Sorcerer, Beneath A Steel Sky and others appearing with either remastered or [...]