Windows 8.1 Repaired

Microsoft fixes its touch interface bug reported previously.

Microsoft Windows 8.1About a month ago, I posted about a bug deep in the Windows 8.1 touch interface code, a problem which triggered an exception in our products.  I detailed the debugging process and verification that it was an error in Microsoft’s latest operating system.

Last week, Microsoft indirectly confirmed the bug by issuing an update, KB2919355, which fixes the reported problem.  With that update installed via Windows Update, our test system now works correctly with all of our unrevised games that had previously responded to a hardware (floating point) exception.  Of course, KB2919355 is a “cumulative update” comprised of more than 100 different fixes so, without unwarranted experimentation, I cannot be certain which one addresses our issue, though KB2927066: Ichitaro crashes when you use a touch screen to enter text in Windows 8.1 seems to be the most likely candidate.  (The description is very similar.)

Unfortunately, we still need to update all of our Windows products, since we cannot rely on customers applying the update (and, ironically, it actually fails to install on one of our development systems).  The fix comes too late, after Windows 8.1 was distributed to the public at large and, also, after I spent many hours debugging the problem.  Still, it is better that it was acknowledged and fixed (than denied and ignored).

A series of updates for Goodsol products will commence shortly.

Debugging Windows 8.1

There is bug deep in the Windows 8.1 touch interface code.

Over the past couple of weeks, I spent quite a lot of time trying to debug an issue that was causing crashes in our programs under Windows 8.1 and which, ultimately, turned out to be a bug in the touch interface code of the operating system.  The problem was not with our code; in fact, our tight programming practices actually outed Microsoft’s failure.

To see how we got to that point, read on…

Symptoms

Late last year, we got a single report of a crash in Pretty Good MahJongg under Windows, occurring only when the touch screen was being used; playing the game with mouse and keyboard worked fine.  Crashes in Pretty Good MahJongg are extremely rare, and this had all the earmarks of a device driver problem, so it was given fairly low priority.  Additionally, we did not have the necessary hardware (at that time) to reproduce the error.  Then we got a second report, and we knew something was amiss.

The first two bug reports were nearly identical, and the obvious commonality (and difference from our systems) was that both machines were running Windows 8.1 and, obviously, had touchscreens.  We had tested our products under Windows 8 when it was released, but did not check Windows 8.1 nor using a touch interface (which, in an ideal world, should not crash programs in any event).  Something appeared to be happening with either Windows 8.1 or the touchscreen interface, or both.

Specifically, the error being reported, in all cases, was “Floating point underflow“, though that actual text for the error comes directly from our exception handler, not the system.

Diagnosis

Our products have a very good crash logging system, so when I got the crash dumps, I discovered that the crashes did not appear to happen in program code at all and, in fact, some of them showed only system routines in the stack dump, while others were essentially the same, but with our message loop (but no other program code) in the stack.

My immediate thought was that the problem could be a message collision, knowing that the touch interface added some new Windows messages.  This could mean that either a touch message was triggering program (e.g., animation) code at an unexpected time, perhaps prior to initialization, or vice versa, with a program message causing driver code to be executed improperly.  This could potentially explain both stack conditions, although I would expect to see our program code elsewhere, but that was never the case.

The bigger problem, at first, was that all of the PGMJ message processing code was identical to that in the Goodsol Solitaire Engine, which drives Goodsol Solitaire 101, Most Popular Solitaire, and FreeCell Plus, as well as the code in Action Solitaire, and that most of the message loop is actually contained in a common library shared by all of these products.  After initial confirmation that all reports were for PGMJ, this concern was finally resolved when crash reports began to escalate, and they expanded to include the whole range of products.  At least the new reports fulfilled my expectation.

Of course, to get to the bottom of the problem, I needed to be able to reproduce the error myself, so I ordered a Windows 8.1 tablet (Dell Venue 8 Pro) for testing.  Fortunately, this tablet displayed the error, and I was able to determine a little bit more about the issue.  The crash happened immediately upon the very first touch within the program, whether clicking a button or simply selecting an edit box, though navigating the program with the virtual keyboard worked…  that is, right up until the first (virtual mouse) touch. 🙁

I built a version of PGMJ that moved custom messages elsewhere in the numbering space, but that made no difference at all.  I tried a couple of other brute force experiments, but nothing altered the crash behavior one bit, so I set up remote debugging on the device and began to debug the program properly.  Unfortunately, the debugger saw the stack in exactly the same way as our exception handler, so every crash was deep in system code and if any program routine was in the stack, it was only the message loop.  Still, our program was definitely and consistently crashing, which meant something was different.  The one major advantage of proper debugging, though, is that I got full symbols, so I was able to determine that the actual crash was happening in ‘ninput.dll‘.  But why?

Here you may imagine days of various attempts at debugging the root cause of the crashes, including “handling” certain messages rather than calling DefWindowProc(), doing the opposite and not processing any messages, and setting breakpoints all over the place and, mostly, being disappointed at how few triggered.  I finally narrowed down the issue to happening from a DialogProc() function within the common library when the (new) WM_GESTURENOTIFY message was posted.  That message is the result of the default processing of a WM_GESTURE message, so presumably handling that in some way would prevent the crash.  No dice.  There is a strange documentation conflict when WM_GESTURENOTIFY is sent to a dialog box, since, “This message should always be bubbled up using the DefWindowProc function.”  However, regarding DialogProc, “Although the dialog box procedure is similar to a window procedure, it must not call the DefWindowProc function to process unwanted messages.”  This gave me a bit of a combinatorial problem, too, but nothing seemed to have any effect on the crash.

Finally, frustrated, I regressed to pure shotgunning of the problem.  I knew that not all programs crashed when touched under Windows 8.1, but (all of) ours consistently did, albeit not in our code.  I added an early message box to demonstrate crashing before any of the dialog boxes or other interface features were shown, and then I began removing pieces of the initialization code.  Voila!  The issue revealed itself!

After removing some of the very first initialization code, executed prior to almost anything else being done, and seemingly entirely unrelated to interface code, the crashes disappeared (though, of course, the program no longer worked).  Methodically reducing the amount of code removed, I was able to determine that the crashes were triggered (but not caused) by three simple lines of code in the exception handler initialization.

Problem

Ultimately, the crash problem was a result of the following C++ code:

    unsigned flags = _controlfp ( 0, 0 );
    flags &= ~( _EM_INVALID | _EM_DENORMAL | _EM_ZERODIVIDE | _EM_OVERFLOW | _EM_UNDERFLOW );
    (void)_controlfp ( flags, MCW_EM );

This, very simply, enables floating point exceptions within the program, including the (now problematic) _EM_UNDERFLOW exception.  The purpose was to provide maximum checking for errors in our code, which is usually so clean that it squeaks.  We never imagined that it would catch errors in a released operating system.  For reference, the above code has been shipping for more than 9 years, to many thousands of customers (and potential customers), and never had any problem before Windows 8.1 arrived.

To be perfectly clear, the actual bug is in Windows 8.1, specifically within ‘ninput.dll’.  There is an error in that module, creating a floating point underflow exception, compounded by reliance on a particular floating point state, namely that the hardware underflow exception is (and remains) disabled.  This is a flaw in the operating system, even though the default floating point state and, therefore, most programs do not display symptoms.

Solution

The actual solution, of course, is to remove the above code, which is a workaround to avoid triggering the crashes.  The tradeoff is that our programs will no longer be quite as robust in detecting floating point errors, but as stated above, this checking has been in place for almost a decade without finding any problems in our code, so it should be fairly safe to remove at this point.

Note that removing these three lines of code is actually more than needs to be done to resolve the immediate problem (i.e., the underflow error exception), but enabling the other exceptions still provides additional places for the operating system to fail, perhaps even further along in the same processing path.  The fundamental problem is that Microsoft counted on the default floating point state (and never tested otherwise) for its latest touch interface code, so it is safest for us to simply revert to using the default state as well.

Verification

It is not enough to simply come up with a solution; that solution must be verified.  We approached this issue in two different ways.

First, I built a new beta version of Pretty Good MahJongg with the above solution applied, and that version was provided to as many of the PGMJ customers who reported a problem as feasible.  Every single one (who reported back) confirmed that the crashes were gone.

Second, we bought a brand new Ultrabook laptop with a touchscreen for testing on a different device.  The laptop shipped with Windows 8 (not 8.1), so it was perfect for conducting our verification tests.

I installed the shipping version of PGMJ (Pretty Good MahJongg 2.41) using nothing but the touch interface, and everything worked fine.  We tested several games and had no problems at all (n.b., under Windows 8).  Then, I upgraded the laptop to Windows 8.1 and confirmed that the crash detailed above happened in exactly the same manner and place when using the touchscreen, but the game was perfectly playable with the mouse and keyboard (until one forgot and touched the screen 🙂 ).  Finally, I installed the beta version of PGMJ with the workaround, and everything worked again; in fact, this is a great way to play the game, especially for a title designed without touchscreens in mind.

Given that we verified the solution using two different and separate processes, we are confident that the issue is resolved.  Indeed, Pretty Good MahJongg 2.5 will be released on March 25, so look for it, still the very best tile matching games available for Windows.

For those who know some of my background, the score now stands as follows:
Gregg Seelhoff 3 – Microsoft 0

MAS Preparation, Part 6

App Sandboxing implementation

In the previous installment of Preparing for Mac App Store Submission, I discussed the reasons and methods for implementating Mac App Store receipt validation within your project.  At this point in the series, you may have already completed all of the currently necessary and recommended changes for converting your Mac OS X game, available for direct download, into an application suitable for the Mac App Store (MAS); you may have even submitted it already.

This final part is a discussion of the latest imposition by Apple, a technology known as application sandboxing, which was introduced with Mac OS X 10.7 Lion.  The general concept of sandboxing is that an application simply runs in its is own “sandbox”, unable to do anything outside its own little piece of the file system, except in specific cases, for which it must have entitlements.  The idea is that a compromised program (via malware or bad programming) is limited in the amount of system damage that can be done.  This is the way that all iOS applications work, and this is, in fact, just a transference of that approach to OS X.  The problem, of course, is that desktop systems are (and should be) different than tablets and phones, so certain types of programs are left out, including those which interact with other programs, such as those which provide ease of access to people who need assistance using computers.  (This is a very poorly considered decision on Apple’s part.)

Frankly, the main reason that you need to understand application sandboxing is that, as of June 1, 2012, all new applications and significant updates submitted to the Mac App Store are required to be sandboxed.  (This deadline was recently pushed back from the original date of March 1 in light of a considerable number of issues that Apple is still having with their implementation.)  If you already have a MAS application (or submit one before the deadline), you will be allowed to apply bug fixes without adding sandboxing, but otherwise, you will need to have sandboxing implemented.  (At least, that is what they are saying right now.)

0. Understand sandboxing

The first step is to understand application sandboxing and the implications of using it for your game.  The definitive documentation for Apple’s implementation of this technology is App Sandbox Design Guide, which is recommended reading.

Essentially, by default, your application is contained entirely within its own (read-only) bundle and a directory named by your bundle identifier (e.g., ‘com.goodsol.com/PrettyGoodSolitaire’) under the ‘~/Library/Containers’ folder.  An internal technology called Powerbox regulates (reads: mostly disallows) any access outside these two folders.  Most access to hardware features and user information is also disallowed without a specific entitlement.

If your game is entirely self-contained, then sandboxing may not present much of a problem.  If you use certain technologies, such as submitting high scores to a web site via the Internet, then you will have to enable appropriate entitlements.  If you have more involved interactions, especially involving other applications (or helpers), you may be forced to redesign/eliminate those features, or else give up on sandboxing (and, hence, the Mac App Store).  Whatever the situation, you will need to determine how this requirement will affect your application.

In our case, we lost the capability to load custom card sets, since the default operation (in the downloadable version) is to read installed files from the ‘Application Support’ folder, though that was not a huge blow since MAS already killed the idea of downloading improvements to the game.  The other feature that is gone is the ability to explicitly load and save games (for now) because the current version of our games are still built with Carbon, which cannot navigate the sandbox/Powerbox.  (Our primary save game feature, AutoSave, still works perfectly fine.)

1. Identify possible sandboxing issues

The next step is to identify possible issues with sandboxing your application, so let’s look at three different categories of features…

What you CAN do:
  • You may read files from your application bundle, as usual.  (Bundles should always be considered read-only, so your code should never attempt to write here.)
  • You may read and write files within your sandbox folders; this process is transparent provided you are using system calls to obtain the correct user folders (and you should never hard-code folder paths, anyway).
  • You may update your application preferences (.plist) using a standard API.
  • You may provide a simple link to a web page; this is allowed by default entitlements (i.e., not considered an outbound connection).
What you CANNOT do:
  • You may not read from nor write to most locations outside your sandbox, including the user desktop and document folders, without the user explicitly selecting the location (and the appropriate entitlement set).
  • You may not use any Carbon navigation dialogs, nor any Cocoa navigation dialogs other than NSOpenPanel and NSSavePanel, and you may not simulate user input to those panels.
  • You may not use “incompatible” functions, including those included in the Authorization Services and Accessibility APIs.
  • You may not send Apple events to arbitrary applications, nor broadcast user-info dictionaries.
  • You may not load any kernel extensions.
  • You may not alter preferences of other applications, nor terminate other applications, including helper applications.
  • You may not sign future updates with a different certificate than the original; doing so will cause Mac OS X to deliberately crash the program.
What requires an entitlement:
  • You need an entitlement to allow outbound network sockets (i.e., for sending information to the internet, operating as a client).
  • You need an entitlement to allow inbound network connections (i.e., for acting as a server).
  • You need an entitlement to allow read or write access to folders or files selected explicitly by the user (via NSOpenPanel/NSSavePanel).
  • You need entitlements to access the user’s ‘Movies’, ‘Music’, and ‘Pictures’ folders.
  • You need entitlements to access camera, microphone, or Bluetooth devices.
  • You need entitlements to access serial, USB, or FireWire ports.
  • You need an entitlement to allow printing.
  • You need entitlements to access the user’s address book or calendars.
  • You need an entitlement to allow the use of Core Location for determining geographical location.

 

After perusing the notes above, write down a list of features that may (or should) fall afoul of application sandboxing and be sure to test each of these features to confirm that they are working prior to enabling sandboxing entitlements.

2. Enable default entitlements

The next step is to enable the default entitlements for sandboxing; essentially, this disables all restricted access types, which allows us to verify which features of the game really need entitlements, and which work without modification.

To do this in Xcode 4.2.1 (under Lion) is fairly simple: Just open the ‘Summary’ tab for your target settings, scroll down to the ‘Entitlements’ section, and check the ‘Enable Entitlements’ (top) checkbox, and then confirm that the ‘Enable App Sandboxing’ checkbox is also selected.  If you are not already using iCloud in your project, then you should make sure that the ‘iCloud Key-Value Store’ is unchecked and cleared, and that that ‘iCloud Containers’ table is empty.

What the first checkbox actually does is create an entitlements file, a property list with the root name from ‘Entitlements File’ (defaults to the project name) and ‘.entitlements’ as the extension (e.g., ‘Pretty Good Solitaire.entitlements’), and it adds that file to your project.  The other (app sandboxing) checkbox automatically adds the ‘com.apple.security.app-sandbox‘ key to the entitlements file, which sets the default/maximum restrictions.

In order to build with application sandboxing enabled, you must be code signing the project.  Clearly this is already the case for projects being submitted to the Mac App Store.  However, if you are only testing and/or do not have an appropriate certificate (such as in our case where, as a contractor, I do not have access to the distribution certificate), you can follow the instructions in the ‘Create a Code Signing Certificate for Testing‘ section of the App Sandbox Quick Start[Editor: Sorry for no direct link, but section links on that page are fuzzled.]  The next section, ‘Specify the Code Signing Identity’, explains how to make the new certificate work with your project.

3. Verify access failures

Your next step is to verify access failures with application sandboxing enabled in the project.  To do this, start by launching the Console utility (from ‘Utilities’) and clearing its display.  This will show log entries to confirm a service denied by sandboxing.

Now, work through the list of suspect features that you created earlier and retest every one of those features with the new signed/sandboxed build.  Failures are, of course, expected, and for every failure there should be a log entry in Console from “sandboxd” and usually containing the text “deny” and an indication of the restricted service.

Note each confirmed failure, and determine which (if any) of the entitlements listed on the target ‘Summary’ page is likely to resolve the issue.  Do not make any changes to the entitlements until all of the suspect features are tested and notated.  A complete list of available entitlements can be found on the Enabling App Sandbox page of the Entitlement Key Reference.  (Note that some entitlements may not be listed on the ‘Summary’ page, requiring direct editing of the entitlements property file.)

If there are any features that are confirmed to fail only under sandboxing and for which there is not an appropriate entitlement, you may want to check the App Sandbox Temporary Exception Entitlements page for a solution that, as the name suggests, may work for the time being, but will probably not be available in the future.  If none of the entitlements nor temporary exceptions will resolve the problem, then you may need to redesign or remove that feature and/or lobby Apple to add an entitlement for the particular action you are attempting.

4. Add necessary entitlements

The next step is adding the necessary entitlements to the project, which should be done as an iterative process.  For each entitlement that you anticipate requiring, enable that entitlement, rebuild the project, then test every feature that is confirmed to have failed with sandboxing previously, noting each one that now functions correctly.

The general idea is that one should minimize the number of entitlements included (requested?) in a sandboxed application.  If you add an entitlement that does not actually help any of your features function, then you should immediately remove that entitlement.  Using excessive entitlements, in addition to reducing the (dubious) security, is a potential cause for rejection from the Mac App Store.  (In other words, simply checking all of the boxes may work in testing, but not for MAS submission.)

When you have completed this process, you should have a reasonably small number of entitlement keys (perhaps even zero) enabled, and all of the features of your program should be functioning properly.  It is always a good idea to double-check all of the identified features, as well as do some additional testing, once the “final” list of entitlements is enabled.  If you still have features that do not work under sandboxing (only), and have no appropriate entitlements, then you may have a decision to make.

In our case, we only had to add the ‘com.apple.security.network.client’ entitlement key, in order to perform submissions to our online high score server, but then we had to make the hard decision to lose the ‘Open…’ and ‘Save…’ features from our store version (for reasons stated previously).

5. Create file migration rules

The penultimate step in the process is creating file migration rules which is simply a property information file that is used only once (on a system) to indicate which existing files should be moved (not copied) into the sandbox.  If you have a new project, without existing users, then you have the luxury of skipping the final two steps.

The rest of us need to create a new property list, named (exactly) “container-migration.plist“, and add it to the project.  This file contains a dictionary key, ‘Move‘, which contains string keys indicating the names of folders and/or individual files to be moved into the equivalent location within the sandbox (or array pairs of strings if you wanted to both move files into the sandbox and relocate them within the relative folder hierarchy, though I cannot think of a good reason why you would).

The specific documentation for the (simple) format of this migration file is in Migrating an App to a Sandbox (which read).  The relevant portion of our migration file is as follows:

<dict>
    <key>Move</key>
    <array>
        <string>${Library}/Pretty Good Solitaire</string>
    </array>
</dict>

This is all it takes to move the entire (unsandboxed) ‘~/Library/Pretty Good Solitaire’ folder into the equivalent sandboxed folder (the unwieldy ‘~/Library/Containers/com.goodsol.PrettyGoodSolitaire/Data/Library/Pretty Good Solitaire’).

Important: Note that the files are moved to the sandbox; they are not copied (and there is no documented process to only copy them).  The ramifications of this are that existing users switching to a sandboxed version will automatically have their files and settings moved for them, but those files and settings will no longer be available to any unsandboxed versions, nor to any other application that may have been accessing these files.

6. Test file migration

The final step is to test the file migration process to assure that your migration file works correctly.

On a system on which you have unsandboxed data for the application in question, manually duplicate the folder and/or files (in ‘Finder’), giving you a backup for additional testing (especially important should the migration fail).  Confirm that the sandbox container folder (i.e., ‘~/Library/Containers/<bundle_id>’) does not exist (and delete it and empty the trash if it does).

Build the application with sandboxing enabled and with the ‘container-migration.plist’ file included in the project.  Run the application.  A container folder should have been created and the specified files/folders moved into appropriate locations within the container’s ‘Data’ folder.

If you need to retest (or if you decide against sandboxing entirely), you can completely delete the container folder for your application (‘<bundle_id>’ and below, not the ‘Containers’ folder) and copy the duplicated data back to its original location.

Conclusion

Following the above process to enable application sandboxing in your game, after having updated your project based on the previous five parts, you should be completely ready to submit to the Mac App Store.  Go for it!  Just keep in mind that the process changes, reviewers vary, and if your project is accepted on its first submission, you are very lucky.  However, if you use the available tools to check your code and your project build, and follow the guidelines in this series of posts, your path should be smoother.

I sincerely hope that you have enjoyed, or at least benefited from, all of these posts.  Comments are definitely appreciated, as are links to this blog, good word of mouth, and US currency in both large and small denominations.

MAS Preparation, Part 5

Mac App Store receipt validation

In the last installment of Preparing for Mac App Store Submission, I discussed the (general) source code modifications that you may need to make in order to convert your downloadable Mac OS X project into one acceptable for the Mac App Store (MAS).

Now, in this fifth part, I will talk about a more specific addition to your source code, functionality for checking the presence and validity of a MAS receipt file.  While this change is not required, and would not cause your submission to be rejected, it is strongly recommend for reasons detailed below.  In fact, we will start there…

0. Consider the reason for receipt validation

Before implementation, it is good to understand, in general terms, the purpose of receipt validation.

When somebody “purchases” an application from the Mac App Store, a receipt file is added to the downloaded bundle authenticating it; this includes free products as well.  However, the level of checking provided by the system itself is minimal.  Soon after the MAS launch, it was discovered that there was a fairly simply method [details deliberately omitted] by which even a simple-minded software thief could circumvent the very basic protections and run stolen MAS applications.

The solution, of course, is to verify the validity of the software receipt for your application, which helps prevent “script kiddies” from easily stealing and distributing your products.  Of course, it is never quite as simple as that, especially since code signatures and encryption are inherently complex, and the program needs to validate a receipt file that has not been created yet (making this code rather tricky to test).

Keep in mind that this validation process is optional, so you can choose not to do it all, to fully validate the receipt file, or to implement only partial checks; the choice is up to you.

The official Apple documentation for this process is in Validating Mac App Store Receipts.

1. Obtain a sample receipt for testing

As mentioned above, your first challenge is that the receipt checking code needs to be able to validate a receipt file from a purchase that is to be made in the future.  To do this, you may want to get a sample receipt for testing.

This used to be fairly easy, as Apple provided a sample receipt, along with the associated validation data, to Mac developers.  Unfortunately, recently (as in, since this series began), the certification signature on the original sample receipt expired and, instead of providing a newer one, Apple has decided to remove the sample receipt entirely in favor of connecting to the iTunes servers to obtain one directly.  Whatever.

Now, the process involves getting the application to initially quit with an exit code of 173, which indicates a failed verification, which is supposed to trigger iTunes to prompt you for your (test) account credentials and then download a valid receipt and insert it into the application bundle, as described in Test During the Development Process.  In practice, this definitively does not work all of the time during development, and the prerequisites are not clear.

What is clear is that the application needs to be signed before Mac OS X (version 10.6.6 or higher) will request iTunes credentials, and that they need to be for a test account.  However, any old test account will apparently not do, and a failure to produce and/or download a receipt displays no error message (just no receipt and only a vague “returnCode:1” in the console).  It appears to be that the application needs to already be in iTunes Connect and that the test account must be associated with that application via the master company account.  Unfortunately, for the moment, this remains as an exercise for the reader.

Another (unproven) approach could be simply borrowing a receipt from any purchased application from the Mac App Store.  You will need to know the exact bundle identifier (fairly easy), version number (also easy), and computer GUID (not quite as easy), but you could use such a receipt exactly like the sample receipt previously provided by Apple.

Important: Do NOT include any receipt files in your project.  Receipts are specific to the application, version, and system, and they are generated and added by the Mac App Store/iTunes; including a receipt in your submission bundle will result in summary rejection of the application.

2. Locate the receipt file

The first step to take in your source code is to locate the receipt file for checking.  In the release version, intended for distribution, the location of the receipt is always within your application bundle at ‘Contents/_MASReceipt/receipt‘.

If you decide to use a sample receipt for testing, you will probably want to define its (different) location, the parameters expected from the sample receipt, and a definition to use for conditional compilation.  In our case, the debug versions define USE_SAMPLE_RECEIPT and the other values (from the old sample receipt) like so:

#ifdef APPSTORE
    #define RECEIPT         "Contents/_MASReceipt/receipt"

    #ifdef _DEBUG
        #define USE_SAMPLE_RECEIPT
    #endif

    #ifdef USE_SAMPLE_RECEIPT
        #define SAMPLE_RECEIPT  "/Users/Gregg/Desktop/receipt"
        #define SAMPLE_BUNDLE   "com.example.SampleApp"
        #define SAMPLE_VERSION  "1.0.2"
        #define SAMPLE_GUID     { 0x00, 0x17, 0xF2, 0xC4, 0xBC, 0xC0 }
    #endif
#endif

Note that we explicitly define different path variables (RECEIPT versus SAMPLE_RECEIPT) and only define the sample parameters when USE_SAMPLE_RECEIPT is defined.  This is additional protection against accidentally using a property intended only for testing in release code.  For the same reasons, we also include this check in the code before the main loop:

#ifndef _DEBUG
#ifdef USE_SAMPLE_RECEIPT
    ErrorMessage ( "Warning!  Sample receipt is being checked in release version." );
#endif
#endif

So, our first verification function is GetReceipt(), which returns the path of the (untouched) receipt file.  The routine obtains the path to the bundle and then generates the path directly to the receipt for later verification, adjusted appropriately if USE_SAMPLE_RECEIPT is defined.  Additionally, just for another sanity check, it also verifies that the actual bundle identifier and short version string match definitions within the code (so we can safely use the definitions later).

3. Check the signature

The next step in validating a receipt is to check the signature to verify that it is properly signed by Apple.  Sample code for doing this (as well as the next two steps) can be found in the Implementation Tips section of the Validating Mac App Store Receipts document, as well as via a nice web search.

Our second verification function is CheckSignature(), which returns a PKCS7* data pointer (and is essentially a fleshed out version of Listing 1-4 in the section linked above).  It opens the receipt file, validates the data types, adds the “Apple Root CA” certificate, and verifies the receipt signature (via PKCS7_verify), then cleans up everything except the receipt pointer, which is returned for use in the next function.

At this point, you know that the receipt file exists, it contains data, and its signature is valid.

4. Verify the receipt contents

The next step is validating a receipt is to verify the receipt contents and confirm that the receipt is intended for your application and the current version number.  The document referenced above gives more details about implementation, but the process is essentially stepping through the objects in the receipt and processing each one appropriately.

There are four types of objects within each receipt that are relevant to the validation process: bundle identifier, bundle version, opaque value, and hash.  In this step, you want to verify that the bundle identifier and bundle version match the hard-coded definitions for your project.  Note that matching against values retrieved from the bundle is not as safe, since the ‘Info.plist’ file could potentially be modified to match an invalid receipt.  Also, you will want to store the bundle identifier, opaque value, and hash object values for checking during the next step.

Our third verification function is VerifyData(), which accepts the PKCS7* data pointer returned by the previous function.  It loops through the objects in the receipt and processes them appropriately, failing if either bundle identifier or bundle version do not match our hard-coded values (IDENTIFIER and VERSION, or SAMPLE_BUNDLE/SAMPLE_VERSION), or if either of these objects is missing from the receipt.  Additionally, it saves the objects necessary for the next step.

At this point, you know that the receipt is intended specifically for the current version of your application.

5. Verify the system hash

The next (and, perhaps, final) step in validating a receipt is to verify the system hash to confirm that the receipt is intended for the particular system on which it is being launched.  This step prevents the outright copying of entire application bundles from one system to another.  Of course, on failure, users are presented with the opportunity to enter their iTunes account information to obtain a valid receipt automatically, so legitimate customers are protected.

The general process for this step involves two parts: obtaining the computer’s GUID and then computing the hash of the GUID, which is then compared to the hash value stored in the last step.  The sample code in the Apple documentation referenced above describes both steps sufficiently (in listings 1-3 and 1-7, respectively).

Our fourth verification function is VerifySystem(), which calls a CopyMacAddress() function to perform the first part of the process, then uses that information to check the system hash.  More specifically, the GUID returned is added to a digest, along with the opaque value and the bundle identifier, and an expected hash is calculated.  The routine verifies that the computed hash length is the same as that of hash and then checks that the contents are identical.

At this point in the process, you now know (to a reasonable degree) that the receipt file exists and is completely valid for that system running the current version of your application.

6. Take extra validation steps

If you are particularly concerned (or paranoid), you can take extra validation steps, such as verifying the certification authorities, double-checking the bundle signature, or (apparently) even sending a receipt back to the App Store for verification.  You can also take steps to obfuscate your code and make it harder for crackers to find and modify the application to circumvent your checks, and Apple provides some suggestions.

However, just taking the general steps documented here will probably eliminate 99% of the problem, and with robust coding, error checking, and testing, theft of the Mac App Store version of your game should be minimal.  (In fact, you will likely lose significantly more money to downward pricing pressures than to piracy, but that is another topic altogether…)

Conclusion

With the addition of receipt validation checking to your project source code, your application should not be subject to simple receipt spoofing, once accepted into the Mac App Store.  In the next (final) installment, Part 6: App Sandboxing implementation, I will discuss the process of preparing for and implementing application sandboxing, as required for the Mac App Store by June 1, 2012 (recently pushed back from March 1st).

MAS Preparation, Part 4

Source code modifications

In the previous installment of Preparing for Mac App Store Submission, I provided some guidelines for application data and resources that should be followed (where applicable) as you transform an existing Mac OS X game project into a project that can be successfully submitted to the Mac App Store (MAS).

In this fourth part, I will describe a number of modifications to the source code of your project that you may want to make to avoid unnecessary MAS rejections and, perhaps, improve your customer support and project maintenance.  These recommendations build on the theme of earlier posts, as most of these changes are due to Apple restrictions against anything having to do with downloading or external sales, including the use of registration codes.

As before, these suggestions are based solely on our experience with our products, so it could help to understand the basics of our sales system.  We have two separate downloadable builds for each Mac OS X product: one is a trial version, with limitations, and the other is a full version, which is unrestricted but requires a customer-specific registration code (as well as the hidden download location).  In order to better understand our customers, each web link in the program adds a very basic tracking code, so we know which version is being used; when online high scores are submitted, we include a hash of the registration code so the scores can be applied to the correct player account.

For this process, we have created a new store version, which is unrestricted (like the full version), includes available extra data (avoiding downloads), and cannot require a registration code.  (Instead, the Mac App Store automatically includes a “receipt” which should be checked, as discussed in the next installment.)  Here are the steps we took…

1. Store game data within user library folder

For the Mac App Store, game data that is not directly created by the user must be stored within the user library (not documents) folder.  Originally, we stored our saved games and statistics in ‘~/Documents/<project name>/<player>’, which made them easily accessible for our customers, but MAS guidelines (and an explicit rejection) required us to relocate such files to ‘~/Library/<project name>/<player>’, where they are less discoverable by users, especially since ‘~/Library’ is now hidden in Finder.  In our case, we actually relocated the files for all SKUs, including a function in the downloadable versions to automatically copy this game data to the new location, to make future customer support more manageable.

Note that, with application sandboxing (as discussed in the final installment), the actual location of the (sandboxed) user library is a significantly longer path name, and the main user documents folder is not accessible by default.  Obviously, the paths given above are illustrative only; one should never hard-code a path in project source code.

2. Remove all registration code via conditionals

Since any sort of registration code is forbidden, you should remove all registration code functionality via conditional compilation.  As part of our Project modifications earlier, we added an APPSTORE preprocessor variable to the store version target (only), which now allows us to remove sections of code with “#ifndef APPSTORE”/”#endif” sections.

In our case, every place in the code where the trial and full versions were different were already denoted with the preprocessor variable ‘DEMO’, so we only needed to look at each such section and decide if it should be like the full version (i.e., no limitations), which was the case most of the time, or like the trial version (i.e., no registration codes).

3. Add fixed internal MAS registration code

Because we used registration codes to differentiate between customers, we needed to add a fixed internal MAS registration code for all store version customers.  In order to replicate the previous functionality for online high score submissions, we needed a way to distinguish among different users without accessing any individually identifiable user information.  Alas, in the end, we had to provide a hash of some system information that was reasonably unique, and simply accept the possibility of a certain number of collisions.  (Thus far, MAS sales have not been high enough to get to levels where collisions are likely, though.)

Honestly, there is a possibility that information from the included MAS receipt could have provided a better (unique) customer identifier, but our implementation (above) was completed before receipt checking was considered.  In this case, that is left as an exercise for the reader (but comments on implementation are encouraged).

4. Update version tracking codes

For the store version, you should update any version tracking codes in your project, using conditional compilation (APPSTORE) where necessary.  This will allow any usage statistics to differentiate between downloadable and MAS customers, as well as from trial version users (i.e., potential customers).

In our case, we simply add a character to the end of link URLs (i.e., as part of a single GET variable) that makes this distinction, so we just added another legal value for MAS purchases.  Additionally, our shortened version numbers are slightly different so (savvy) players can identify the product played on the online high score pages (for example, on this FreeCell scores page, see the ‘Product’ column).

5. Remove or modify marketing screens

Consider whether you should modify or remove any marketing screens in your product.  If you have dialog boxes which have the occasional link to ordering pages or downloadable content, you will need to remove those links (and any referencing text); if you have dialogs that are explicitly for the purpose of marketing, it may make sense to remove them entirely instead.

In the case of our products, we include a fairly self-explanatory ‘Help Center‘ screen which not only contains links to the help files for the product, but also serves as a portal to all manner of support services, including sales (verboten), registration code entry (not allowed), customer support email (probably OK), and suggestions for other games (who knows?).  We found that removing the disallowed and questionable content from these pages made them almost pointless, so we instead replaced the entire dialog with a direct link to the help file contents page (within the bundle).

6. Remove links to any downloadable content

You must remove any links to downloadable content or risk the rejection of your product.  Note that these links (and, in truth, a number of other “offenses”) may not always be obvious on initial review, so your product can certainly get accepted once and then later rejected for something that was there in the first version; it has happened to us more than once.  (All Apple reviewers are not created equal.)

In the case of Pretty Good Solitaire, we have card sets which customers can freely download, plus a feature to check for and download updates, both of which are forbidden in the Mac App Store.  The following subsections describe the issues that we had to address to prevent suggesting that there was anything outside the MAS playground.

6a. Remove download links from main menus

We had to remove our ‘Download Additional Card Sets‘ menu option from both the main and game menus.

6b. Remove version checking from main menus

We had to remove our ‘Download Latest Version‘ menu option, which includes an implicit version check, from the both the main and game menus.

6c. Remove download links from submenus

We had to also remove a ‘Download Additional Card Sets‘ menu option from our submenu that players use to select the desired card set (a logical and convenient place to find it).

6d. Remove download links from buttons

Finally, we had a ‘Download Additional Card Sets‘ button within the program preferences, which also had to be removed (actually, hidden).

Note:  In all of these cases, rather than modify our user interface (.nib) files, and thus have to manage different sets of files, we added conditional code that simply removed the offending menu option or hid the button, keeping the source code (including .nib files) identical between downloadable and store versions (distinguished solely by the presence/absence of APPSTORE).

7. Review any use of the Application Support folder

Before submitting your application to MAS, carefully review your use of the Application Support folder (if any).  Specifically, you are allowed to create and (appropriately) use a folder at ‘~/Library/Application Support/<app-identifier>’, where <app-identifier> “can be your application’s bundle identifier, its name, or your company’s name.”

However, there is a twist:  “These strings must match what you provided through iTunes Connect for this application.”  This particular requirement is somewhat buried in the guidelines, and it is not always checked.

In our case, we inadvertently created but did not use (in the store version) a folder, ‘~/Library/Application Support/Goodsol’, which seems to fit the criteria.  However, it was supposed to end in ‘Goodsol Development’ to meet the letter of this requirement.  This “violation” was not discovered until our third or fourth submission, and our downloadable versions already used the folder extensively as named.  Fortunately, the store version did not actually use it (by virtue of including all downloadable content), so we just removed the spurious creation of an empty folder.  Problem solved (albeit with yet another rejection and delay).

Conclusion

With the above modifications to source code, plus perhaps a little detective work finding similar issues with any particular application, your project should be tentatively ready for submission to the Mac App Store.  In the next installment, Part 5: Mac App Store receipt validation, I will discuss the reasons that you should probably validate receipts in your shipping product and provide a few links to useful code and resources.

MAS Preparation, Part 3

Data and Resource guidelines

In the previous installment of Preparing for Mac App Store Submission, I detailed changes to the information property list that are necessary (or recommended) for a Mac OS X game project being converted to one suitable for MAS submission.

This third part deals with practical implications of guidelines for project data and resources, in particular, artwork and help files.  These points are necessarily somewhat specific to the manner in which we implemented our products, but they should give you some idea of items to look out for, and as always, comments on your experiences are encouraged.

Now, please review all of the artwork and other (non-code) resources in your project with these points in mind…

1. Include a 512×512 image in the application icon

The Mac App Store requires a 512×512 image in the application icon file.  Although this is generally overkill for an application, it is necessary for MAS submissions, where that image size is (presumably) used for presenting your product to users.

We actually produce our Macintosh (.icns) icons under Windows using Axialis IconWorkshop, which handles the required sizes without problem; we recommend this tool for processing icons for multiple platforms.  (We have no affiliation with Axialis, except as a satisfied customer.)

2. Update branding artwork

Although it seems (and truly is) silly, you need to review your branding artwork and update anything that even hints that there is another way to obtain software than through the Mac App Store.

Herein lies a theme: Apple hates downloads; Apple hates external sales.  Any suggestion that a customer may download or purchase anything outside of MAS will result in the rejection of your submission.  (Trust me, we had this thrown in our face, via rejection, more than once.)

In our case, we had a static bitmap used as a splash screen.  It included the (innocuous) line, “Download the latest version of Pretty Good Solitaire from http://goodsol.com“; it was hardly noticeable, and the web address was not even a hot link.  Rejection.  We had to remove that text before resubmitting (and we also removed the line with our support email address, just in case).

3. Add any extra product data desired

Continuing with the theme, you will need to add any extra product data that you may want customers to have, especially any data previously available via download.  In our case, we have (20) extra card sets that are available for Pretty Good Solitaire customers to download (and, likewise, extra tile sets and custom layouts for Pretty Good MahJongg), so these needed to be added to the main bundle to avoid Apple’s allergy to downloading.

The one minor advantage of this, of course, is that Apple has to bear the full download bandwidth burden.  (Card set artwork is not small.)

4. Remove trial version resources from the project

If appropriate, remove any resources used only in trial versions of your product.  In our case, we have an exit screen for trial users, as well as a separate interface (nib) file with all dialogs displayed (only) in the trial version, so we removed these.  (There is no sense including unused files, especially if they could give Apple more places to find reasons to reject.)

Similarly, if you have an install bitmap in your bundle, as I recommended in Making Mac Disk Images Pretty, you may as well remove that, too (since there is no disk image packaging to be done).

5. Remove any ‘How to Order’ section from help files

Review your help files and remove any pages that discuss verboten topics, such as downloading, registering, or (especially) purchasing the product.  We have an entire section called “How to Order”, dealing with all of those topics, that is just removed wholesale for the MAS submission (but remains in our downloadable versions); it was the cause for a rejection.

Note that Apple does not reject for simple web links back to a product site, which itself may have loads of sales and download information, but you will want to be careful about the surrounding text.

6. Remove text referring to downloading or sales

Building from the previous guideline, after removing entire pages about problematic topics, check other help topics and remove any text referring to downloads or sales, even indirectly.

In our case, we have rule pages for our bonus games, which are provided in the full (purchased) version, and we removed the line, “Note: This game is only available in the full version of Pretty Good Solitaire.”  Elsewhere, we also replaced “the full version” with “this version”.

Conclusion

Using the above guidelines, you should be able to make sure that your project does not include any content that Apple may find “objectionable”, giving them cause to reject the submission.  In the next installment, Part 4: Source code modifications, I will discuss the numerous (small) changes that may be necessary in your project source code.

MAS Preparation, Part 2

Property List (Info.plist) changes

In the last installment of Preparing for Mac App Store Submission, I discussed the project modifications that are necessary (or recommended) for converting an existing Mac OS X project to one suitable for MAS submission.

This second part describes the changes to the information property list for your application that you should make for successful submission and to eschew rejections for simple issues.  As before, comments about any other issues or different experiences are certainly welcome.

Open the application information property list, usually named ‘Info.plist‘, in your project and follow these steps…

1. Update the bundle version format

First, update the format of the ‘Bundle version’ entry (CFBundleVersion) to contain exactly 3 period-separated integers representing the version number (e.g., “1.01.1“).  It cannot contain alphabetic characters (and although some documentation suggests that it may contain more or fewer integers, we did not take that chance).

For non-MAS applications, the format of this field was not enforced and, in fact, the default ‘About’ box encouraged the use of this field as a standard version description (alphanumeric string) by directly displaying it underneath the application name.  We used a format like, “1.01 (January 2012)”, which is more useful and aesthetically pleasing, but this was cause for rejection.

2. Add an application category

If you do not have one already, you will need to add an ‘Application Category’ entry (LSApplicationCategoryType).  The easiest way to set this value (in Xcode 4) is to select the ‘Summary’ tab for your target and select the appropriate setting in the ‘Application Category’ box.  In the case of Pretty Good Solitaire, we chose ‘Games – Card Games’ (public.app-category.card-games).

Note that Xcode 3 did not recognize this key, so it was necessary to explicitly add it, along with the appropriate category value, as found in the Information Property List Key Reference; fortunately, this is no longer necessary.

3. Set the minimum system version

Next, set the ‘Minimum system version’ entry (LSMinimumSystemVersion) to “10.6.6”, or higher if appropriate.

The Mac App Store does not work on versions of Mac OS X prior to 10.6.6 anyway, and leaving this at a lower setting (even if your downloadable versions support Leopard, Tiger, or even an earlier OS) may be cause for a rejection.

4. Review the supported document types

Finally, review the supported ‘Document types’ entry (CFBundleDocumentTypes), if any.  Remove any document types that will not be supported in your store version.

In our case, we supported a document type for saved games, which is still useful, but also a document type for automatic installation of extra card sets (which can be downloaded), or in the case of Pretty Good MahJongg, types for both extra tile sets and tile matching layouts.  Since downloading or installing any improvement to your application (outside of the Mac App Store) is verboten, we needed to remove this support.

Note that this initially passed muster with our first product, but it was cause for a rejection later on a different product using essentially identical functionality.  There is clearly some subjectivity to the application reviews, so a more thorough (or nitpicky) reviewer may find something previously allowed.  When a rejection can set your release schedule back a couple of weeks or more, it is not worth the risk.

Conclusion

With just these few changes to your information property list, your project should have the necessary application information for submission.  In the next installment, Part 3: Data and Resource guidelines, I will describe the issues you may encounter with your application data.

MAS Preparation, Part 1

Project modifications

In Preparing for Mac App Store Submission, the first set of changes you should make are to the project itself.

This installment describes the project modifications that need to be made to an existing Mac OS X game project.  The general assumptions are that the existing project is working and properly tested, and that, ultimately, you will want to maintain a single set of source code with conditional compilation to differentiate the store version from other builds.  Note that these are the steps that we took for Pretty Good Solitaire Mac Edition; your project may require some adjustments to these steps.  (Comments where any significant change is necessary would be very appreciated.)

So, without further ado…

0. Create a duplicate store project

Before making any other changes, create a duplicate copy of the entire project folder and name the copy appropriately.  In our case, the original folder was ‘Pretty Good Solitaire’ (which builds full and trial versions of the game) and we created a separate ‘pgsse’ (Pretty Good Solitaire [Mac/]Store Edition) folder for MAS modifications.

The version of your project that you will submit to the Mac App Store is a separate SKU (Shelf Keeping Unit), a build that uses slightly different code and configuration and which is distributed via a different channel.  While it is possible to have the store target within the main project, certain features (e.g., Power PC support, Mac OS X 10.5 support, downloadable data) are not supported in MAS, so it is easier to separate them (at first, anyway).  In any event, doing so at the start gives you a safe playground for making changes without messing up your working project.

1. Update the project version number

As just noted, the store build is a separate version and, thus, should have a different version number.  In our case, we decided that odd minor versions (e.g., “1.01”) would represent the store editions, while even minor versions (e.g., “1.00”) represented the direct downloadable editions.

2. Rename the primary build target

In the new project (of course), rename the primary build target to something appropriate.  In our case, we renamed the full version target to ‘Pretty Good Solitaire store’, which is now the store version.  You could also delete any redundant or obsolete targets remaining in the project.  For example, we deleted the trial version target, as it is built in the original project and the MAS version cannot have any vestiges of a trial version.

Note that this step is not strictly required, but it is a good idea to differentiate between targets, so it is always immediately obvious which project is active, and also to minimize any excess baggage, which reduces the possibility of mistakes.

3. Build with a current Xcode version

Make sure that you build the project with a current version of Xcode.  Submissions to the Mac App Store now require Xcode 4 and, as of this writing, the latest version is Xcode 4.2.1.

If you have an older version of Xcode, this would be a good time to upgrade, and although it will not be mentioned explicitly, you should build the project regularly, ideally after every change, to make sure that the build is not broken and behaves as expected.

4. Define a preprocessor variable

To allow for conditional compilation of certain source code, it is a good idea to define a preprocessor variableAPPSTORE, for any build targets intended for MAS.  In the ‘Build Settings’ for the target (or the whole project, if you prefer), find the setting for ‘Preprocessor Macros’ and add “APPSTORE” to each configuration.  Note that it is common to have different variables for ‘Release’ and ‘Debug’ configurations, so be careful to define APPSTORE for each one without accidentally removing or altering any existing definitions.

Of course, there is no requirement that the preprocessor variable be named “APPSTORE”, but beware that simply using “STORE” results in a naming conflict in the latest Mac OS X SDK.

5. Add necessary frameworks/libraries

In order to test the validity of app receipts, you will need to add the IOKit and Security frameworks and the crypto library.  From the ‘Build Phases’ of the primary target, open the ‘Link Binary With Libraries’ section and, by clicking on the ‘+’ symbol, add ‘IOKit.framework‘, ‘Security.framework‘ and ‘libcrypto.dylib‘.

Note that Xcode 4 adds these frameworks/libraries at the top level of your project; you will probably want to drag them into the ‘External Framework and Libraries’ folder with the other frameworks.

6. Configure debugging symbols

Despite submitting a release version to MAS, Apple requires debugging information to be included with a submission.  To accede to this requirement, you must set ‘Generate Debug Symbols’ to “Yes” and also set ‘Debug Information Format’ to “DWARF with dSYM File” (at least for the ‘Release’ configuration) in the ‘Build Settings’ of the target.

In our original project, we had all debug information disabled and/or stripped from the release builds, but one of our early issues was the lack of the dSYM file with debugging information for Apple.

7. Set correct build architecture

Finally, set ‘Architectures’ to an Intel (only) setting; for our project, that is “32-bit Intel”.

Even if your code and original project supports both PPC and Intel via a “Universal” application, the presence of a PPC build in your submission will result in rejection.  (We found that out the hard/lengthy way.)  At least now there are settings for this; in Xcode 3, you had to use “i386”, which was not even listed as a choice.

Conclusion

At this point, you should have a new project with a store target and all of the build settings configured appropriately.  In the next installment, Part 2: Property List (Info.plist) changes, we will discuss the necessary adjustments and additions to the information property list for your project.

Preparing for Mac App Store Submission

Making a Mac OS X game project suitable for MAS

If you currently have a Mac product and have not already done so, you may be considering submission to the Mac App Store (MAS).

In the upcoming series of posts, I will be detailing the process that we went through to get Pretty Good Solitaire Mac Edition, and some of our other game products, successfully submitted to MAS.  There were a number of rejections along the way, as the App Store Review Guidelines [note: requires Mac developer agreement], while extensive, are not comprehensive (nor are they 100% consistent, as we had some products accepted and others rejected with identical behaviors).

Over multiple submissions, and fewer rejections, we developed a submission checklist which I will detail roughly (some items are specific to our games) in these upcoming posts:

We have had product in the Mac App Store since launch day, more than a year ago.  If you already have a game that runs on Mac OS X, it makes sense to make the several modifications to get it into MAS, another channel to find customers.  However, in our experience, it is not a viable substitute for direct downloadable sales.  The channel is not (yet) the primary ‘go to’ location for Mac software, although the availability of Lion (Mac OS X 10.7) only on MAS should shift more customers.  Additionally, there is the same downward pressure on pricing (towards free) seen on the iOS App Store, sales are lackluster, and (of course) you are giving 30% directly to Apple.

I would certainly not recommend developing a project solely for the Mac App Store, nor eliminating a direct downloadable sales channel in favor of MAS, but with an existing project it may be worth the fairly limited extra effort it takes to be there, too.

Accidentally Cheating at Backgammon

Why players perceive unfairness in Backgammon software

On a regular basis, inexperienced Backgammon players voice opinions about how a certain computer program or game server cheats.  The stochastic nature of the game lends itself to this kind of perception on the part of human beings.  Generally, there are a few primary reasons for this type of belief.

First, novice players often fail to recognize the complexities of the game of Backgammon, so what they perceive as an unnatural number of “lucky rolls” are not (necessarily) due to luck, but rather due to skillful play on the part of the opponent.  Expert players tend toward positions where a greater number of rolls would be considered good (i.e., “lucky”).  A higher percentage of good moves tends to make the dice appear biased in ones favor, and it is also key to good checker play.

In many cases, players also fail to understand the nature of truly random numbers.  It is often stated that, say, a certain number of doubles in a row indicates…  excuse me…  “proves” that the virtual dice are unfair when, in fact, a truly random number generator would have to produce any arbitrary sequence (whether or not a pattern is perceptible) given enough rolls.  Of course, we are talking about pseudo-random number generators (PRNG), so they are, by their very nature, not truly random.  However, one would have to do an actual study/count of the dice rolls to make any conclusion about any particular PRNG.

The reason for this need to analyze a PRNG scientifically, rather than anecdotally, seems fairly obvious.  Human beings have selective memory, which means that we tend to recall things that are out of the ordinary, so a number of doubles in a row stands out, whereas a statistically identical sequence of rolls that do not seem to show a pattern are not reported.  Likewise, a few very good (or very bad) rolls are more memorable than many run-of-the-mill rolls.

Related to this is the concept of apophenia, which is the human “experience of seeing patterns or connections in random or meaningless data.” [from Wikipedia]  Our minds have evolved to recognize patterns, so we can sometimes perceive things that are not there.  This is how people see images in clouds, hear music or sounds in white noise, and imagine divine imagery in oil stains or burnt toast.

All of these factors make it very easy for an average person to perceive unfairness in Backgammon software or servers (even in games against other human beings), and even trained experts can be fooled.

How experts demonstrate that Backgammon software is fair

There are a few key points that are usually made by experts when arguing that a particular Backgammon program does not cheat.  First, of course, one generally describes some of the aspects of the perception problem, as listed above.  In particular, reports are almost always anecdotal, so they can be dismissed quickly as having no scientific validity until somebody does an actual count and statistical analysis.

To dismiss accusations of manipulated dice (by software), the suggestion is to manually input dice rolls, which most (decent) programs allow, according to the rolls of physical dice recorded meticulously, or by changing to an alternative PRNG.  If the results stay statistically consistent, that argues against the idea that the rolls are manipulated.  Another common argument is that programs can “look ahead” to see which rolls are upcoming and make moves based on this prior knowledge, and manual input of dice rolls also removes this possibility.

Another method to test if dice rolls are being artificially manipulated is to switch sides and look for discrepancies.  In other words, start a game (or save one in progress) with a particular random number seed and play the rest of the game, recording the dice rolls for each side.  Then, restart (or load) the game and play the opposite side.  If the dice rolls remain the same, then no manipulation was done to bias the outcome.

A final, less scientific, approach is the simple “Why?” method, wherein one looks at the reasons why (and how) a programmer might decide to write a biased program.  Speaking as the primary programmer for MVP Backgammon Professional, from MVP Software, I can assure you that cheating would add a whole extra layer of (unwanted and unnecessary) complexity, so I certainly did not and would not include such code.  In fact, accusations of unfairness were troubling enough to MVP for the first version of the program that our version has a replaceable PRNG library so one can write ones own (with whatever extra checking is desired).

Possibility for Backgammon software to cheat without malice aforethought

This whole topic was reinvigorated when yet another thread appeared on rec.games.backgammon recently, entitled “Jellyfish.  Cheating or just Lucky” [links to Google groups].  Through dozens of messages, some people suggested/argued that the Backgammon program Jellyfish seemed to cheat, while two other popular programs, GNU Backgammon and Snowie, did not.

Interestingly (and, n.b., anecdotally), when testing MVP Backgammon, I had a similar experience.  I was simply testing relative strength with a series of 25-point matches between my program and these others.  Whereas the strength of my neural network was comparable to the others, it got beaten significantly by Jellyfish when it rolled the dice.  When MVPBG rolled, it was much closer.  As a final test, I played one match with manual rolls, and it was again close.  At this point, I figured out the likely problem (leaving alive the possibility that it was just sheer chance).

The whole purpose of a neural network is to discover connections and patterns in provided data, and the conclusions are affected by the design of the inputs (essentially, which raw data is supplied) and, of course, the requested output(s).  In our design, we basically supplied the number of checkers on each point (in a special format), the number on the bar, and the number borne off.  This specifies a pure position in the game (with no knowledge about moves or rolls), and our outputs were designed to estimate the probability of each potential game outcome (win, loss, or winning/losing either a gammon or backgammon).  The neural network was only used for evaluation; the selection of moves was based on the evaluation of the resulting position (and cube decisions were calculated mathematically from the neural network outputs).

Theoretically, we could provide irrelevant inputs (e.g., outside temperature) and during training, their influence on the network would tend toward zero.  However, providing somewhat related data, such as the last game move, could give the neural network just enough information to begin to anticipate an outcome and bias the outputs.  More directly, providing the current dice roll, or perhaps designing the neural network to rate individual moves based on that roll, gives the network additional information that could be used to actually predict the next pseudo-random roll, especially if the particular PRNG is not very good.  After all, guessing what the next roll would be based on the position and previous roll is exactly the kind of task that neural networks are designed to solve.

Based on this observation, I suggest that it is possible that the programmers of Jellyfish may have inadvertently, and with no malicious intent whatsoever, provided their neural network with just a little too much information, and it may have taken that information to (at least partially) figure out the random number sequence and then draw conclusions that were not intended.

This would be a very interesting (and perhaps slightly startling) example of emergent behavior in a computer system.  It would, however, explain why a program could pass all of the tests to “prove” it is not cheating, but still have an observable bias when using its own dice.  I suppose we could call it “computer intuition“.  Of course, without more scientific study, it could still just be called “luck“.