Monday, June 15, 2009

Online Hard Problems

I've tried to keep a list of "hard problems" for online games. Sometimes it is to impress folks with how much work is involved in trying to build the technology itself (and redirect them toward middleware), and sometimes it is to remind me of how much work we have left as a middleware company. Here is my current collection. It is not every system needed, but the ones that are particularly tricky or challenging and require the right architectural decisions early. I'd appreciate if folks would add to this.

Hopefully, over time I'll be able to publish my conclusions about how to actually solve these problems. Many interact, and need a unified approach to be able to be solved at all.

Distributed and parallel nature
  • Load balancing.
  • Latency hiding.
  • Timing, synchronization and ordering.
  • Anticheating/security
  • Threading,
  • Execution performance,
  • Memory performance,
  • Network performance and message streaming,
  • Persistence performance,
  • Bandwidth reduction,
  • Content streaming/background loading
Scale and seamlessness
  • Scale for both casual and intensive styles (reusing the same infrastructure)
  • Seamlessness at runtime
  • Entity interaction across server "boundaries"
  • Seamlessness at edit time
  • Transactional entity interaction
  • Shardless worlds
Ease of Use/Development/Operations
  • Ease of use for designers, game developers (code and assets)
  • Ease of developing large scale content
  • Development efficiency and rapid iteration (assets and code),
  • Fault tolerance, debuggability of large distributed system,
  • Performance debugging,
  • Script/behavior debugging, script/behavior ease of use,
  • Hosting, operations ease of use (server deployment, control, monitoring, ...)
  • In game microtransactions (security)
  • Integration to multiple platform services
  • Client deployment, patching, binary patching


  1. Nice list! Some of those little bullets represent SO much effort... This is a fun topic to think about.

    What is "multiple platform services"? Is that stuff like XBox Live? All the unique billing solutions out there?

    "Persistence performance": I presume you don't just mean writing good database code, but also the granularity of persistence (every property change? Just when the character logs out?)

    Some candidate bullet points:

    Interaction with source control and branching.

    Versioning persisted assets.

    Activity logging (for design review, exploit hunting, customer service, etc)

  2. What I mean by platform services is billing, account management... Each publisher may have their own system you have to integrate to.

    There is a really important trade off between performance, granularity of persistence, consistent transactions between entities (especially between hosts), lost play time and consistency in the face of crashing.

    Agree about metrics, terms-of-service logging. There is so much data to consume and analyze. Game design exploits are really hard to find "automatically". It may be cheaper to resort to "social engineering".

    Concurrent content editing is another hard one with multiple solutions to choose from.

  3. I recently came accross your blog and have been reading along. I thought I would leave my first comment. I dont know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.


  4. I am not so good about technical aspect. I am the usual player of Download Games/Video Games that you do. And I appreciate your every effort to make us learn and have fun. Hope every developer continuously improve more and more games.