Online Hard Problems

I've tried to keep a list of "hard problems" for online games. Sometimes it is to impress folks with how much work is involved in trying to build the technology itself (and redirect them toward middleware), and sometimes it is to remind me of how much work we have left as a middleware company. Here is my current collection. It is not every system needed, but the ones that are particularly tricky or challenging and require the right architectural decisions early. I'd appreciate if folks would add to this.

Hopefully, over time I'll be able to publish my conclusions about how to actually solve these problems. Many interact, and need a unified approach to be able to be solved at all.

Distributed and parallel nature

• Load balancing.
• Latency hiding.
  
• Timing, synchronization and ordering.
• Anticheating/security

Performance

• Threading,
  
• Execution performance,
• Memory performance,
  
• Network performance and message streaming,
• Persistence performance,
• Bandwidth reduction,
• Content streaming/background loading

Scale and seamlessness

• Scale for both casual and intensive styles (reusing the same infrastructure)
  
• Seamlessness at runtime
  
• Entity interaction across server "boundaries"
  
• Seamlessness at edit time
• Transactional entity interaction
• Shardless worlds
  

Ease of Use/Development/Operations

• Ease of use for designers, game developers (code and assets)
  
• Ease of developing large scale content
• Development efficiency and rapid iteration (assets and code),
• Fault tolerance, debuggability of large distributed system,
• Performance debugging,
• Script/behavior debugging, script/behavior ease of use,
• Hosting, operations ease of use (server deployment, control, monitoring, ...)
• In game microtransactions (security)
• Integration to multiple platform services
• Client deployment, patching, binary patching