Failing In So Many Ways


Liang Nuren – Failing In So Many Ways

Python JSON Performance

So I’ve been pretty open about the fact that I’ve moved from data warehousing in the television and online ad industries to data warehousing in the gaming industry. The problem domains are so incredibly different. In the television and ad industries, there’s a relatively small amount of data that people are actually concerned about. Generally speaking, those industries are most interested in how many people saw something (viewed the ad), how many people interacted with it (clicked on it), and whether they went on to perform some other action (like buying a product).

However, in the gaming industry we’re interested in literally everything that a user does – and not in the creepy way. The primary goals are to monitor and improve user engagement, user enjoyment, and core business KPIs.  There are a lot of specific points to focus on and try to gather this information, and right now the industry standard appears to be a highly generalized event/payload system.

When looking at highly successful games like Temple Run (7M DAU [gamesbrief]) it’s only 150 events per user to get a billion events per day.  Between user segmentation and calculating different metrics it’s pretty easy to see why you’d have to process parts of the data enough times that you’re processing trillions of events and hundreds of GB of facts per day.

When I see something that looks that outrageous, I tend to ask myself whether that’s really the problem to be solving. The obvious answer is to gather less data but that’s exactly the opposite of what’s really needed. So is there a way that to get the needed answers without processing trillions of events per day? Yes I’d say that there is; but perhaps not with the highly generic uncorrelated event/payload system.  Any move in that direction would be moving off into technically uncharted territory – though not wholly uncharted for me. I’ve built a similar system before in another industry, albeit with much simpler data.

If you aren’t familiar at all with data warehousing, a ten thousand foot overview (slightly adapted for use in gaming) would look something like this.  First, the gaming client determines what are interesting facts to collect about user behavior and game performance. Then it transmit JSON events back to a server for logging and processing.  From there the data is generally batch processed and uploaded to a database* for viewing.

So as a basic sanity check, I’m doing some load testing to determine whether it is feasible to gather and process much higher resolution information about a massively successful game and it’s users than seems to be currently available in the industry.  Without going into proprietary details, I’ve manufactured analytics for a fake totalhelldeath game.  It marries Temple Run’s peak performance with a complicated economy resembling Eve Online’s.

From there, I’m compressing days of playtime into minutes and expanding the user base to be everyone with a registered credit card in the app store (~400M people as of 2012) [wikipedia].  The goal here is to see how far it’s possible to reasonably push an analytics platform in terms of metrics collection, processing, and reporting.  My best estimate for the amount of data to be processed per day in this load test is ~365 GB/day of uncompressed JSON.  While there’s still a lot that’s up in the air about this, I can share how dramatically the design requirements differ:


  • Reporting Platform: Custom reporting layer querying 12TB PostgreSQL reporting databases
  • Hardware: Bare metal processing cluster with bare metal databases
  • Input Data: ~51GB/day uncompressed binary (~150TB total uncompressed data store)
  • Processing throughput: 86.4 billion facts/day across 40 cores (1M facts/sec)

Analytics Load Test:

  • Reporting Platform: Reporting databases with generic reporting tool
  • Hardware: Amazon Instances
  • Input Data: ~365 GB/day uncompressed JSON (~40k per “hell fact” – detailed below)
  • Processing throughput: duplication factor * 8.5M facts/game day (100 * duplication facts/sec)

I’ve traditionally worked in a small team on products that have been established for years.  I have to admit that it’s a very different experience to be tasked with building literally everything from the ground up – from largely deciding what analytics points are reasonable to collect to building the system to extract and process it all. Furthermore, I don’t have years to put a perfect system into place, and I’m only one guy trying to one up the work of an entire industry.  The speed that I can develop at is critical: so maintaining Agile practices [wikipedia], successful iterations [wikipedia], and even the language I choose to develop in is of critical importance.

The primary motivator for my language choice was a combination of how quickly I can crank out high quality code and how well that code will perform.  Thus, my earlier blog post [blog] on language performance played a pretty significant role in which languages saw a prototype.  Python (and pypy specifically) seems well suited for the job and it’s the direction I’m moving forward with.  For now I’m building the simplest thing that could possibly work and hoping that the Pypy JIT will alleviate any immediate performance shortfalls.  And while I know that a JIT is basically a black box and you can’t guarantee performance, the problem space showed high suitability to JIT in the prototyping phase.  I foresee absolutely no problems handling the analytics for a 1M DAU game with Python – certainly not at the data resolution the industry is currently collecting.

But, I’m always on the look out for obvious performance bottlenecks.  That’s why I noticed something peculiar when I was building out some sample data a couple of days ago. On the previous project I worked on, I found that gzipping the output files in memory before writing to disk actually provided a large performance benefit because it wrote 10x less data to disk.  This shifted our application from being IO bound to being CPU bound and increased the throughput by several hundred percent.  I expected this to be even more true in a system attempting to process ~365GB of JSON per day, so I was quite surprised to find that enabling in-memory gzip cut overall application performance in half.  The implication here is that the application is already CPU bound.

It didn’t take much time before I’d narrowed down the primary culprit: json serialization in pypy was just painfully slow. It was a little bit surprising considering this page [] cites pypy’s superior json performance over cpython.  Pypy is still a net win despite the poor JSON serialization performance, but the win isn’t nearly as big as I’d like it to be. So after a bit of research I found several json libraries to test and had several ideas for how the project was going to fall out from here:

  • Use a different json library. Ideally it JITs better than built in and I can just keep going.
  • Accept pypy’s slow json serialization as a cost of (much) faster aggregation.
  • Accept cpython’s slower aggregation and optimize aggregation with Cython or a C extension later
  • Abandon JSON altogether and go with a different object serialization method (protobuf? xdr?)

After some consideration, I ruled out the idea of abandoning JSON altogether. By using JSON, I’m (potentially) able to import individual records at any level into a Mongo cluster and perform ad hoc queries. This is a very non-trivial benefit to just throw away! I looked at trying many JSON libraries, but ultimately settled on these three for various reasons (mostly relating to them working):

To test each of these libraries, I devised a simple test with the goal of having the modules serialize mock event data.  This is important because many benchmarks I’ve seen are built around very small contrived json structures.  I came up with the following devious plan in order to make sure that my code couldn’t really muck up the benhmark results:

  • create JSON encodable dummy totalhelldeath fact list
  • foreach module: dump list to file (module.dump(facts, fp))
  • foreach module: read list from file (facts = module.load(fp))

Just so that everything is immediately obvious: this was run on one core of an Amazon XL instance, and the charts are measuring facts serialized per second.  That means that bigger bars are better here.

Read Performance

There’s really no obvious stand out winner here, but it’s obvious that the builtin json library is lacking in both cpython and pypy. It obviously runs a bit faster with cpython, but it’s not enough to really write home about. However, simplejson and ujson really show that their performance is worth it. In my not-so-expert opinion, I’d say that ujson walks away with a slight victory here.

Write Performance

However, here there is an obvious standout winner. And in fact, the margin of victory is so large that I feel I’d be remiss if I didn’t say I checked file sizes to ensure it was actually serializing what I thought it was! There was a smallish file size difference (~8%), primarily coming from the fact that ujson serializes compact by default.

So now I’m left with a conundrum: ujson performance is mighty swell, and that can directly translate to dollars saved.  In this totalhelldeath situation, I could be sacrificing as much as 71k + 44k extra core-seconds per day by choosing Pypy over CPython.  In relative money terms, that means it effectively increases the cost of an Amazon XL instance by a third.  In absolute terms, it costs somewhere between $5.50 USD/day and $16 USD/day – depending on whether or not it’s necessary to spin up an extra instance or not.

Obviously food for thought. Obviously this load test isn’t going to finish by itself, so I’m putting Python’s (lack of) JSON performance behind me.  But the stand out performance from ujson’s write speed does mean that I’m going to be paying a lot closer attention to whether or not I should be pushing towards CPython, Cython, and Numpy instead of Pypy.  In the end I may have no choice but to ditch Pypy altogether – something that would make me a sad panda indeed.


Filed under: Data Warehousing, Game Design, Personal Life, Software Development

CSM Minutes: Misleading

Let me begin with a quote from the CSM minutes:

CCP Greyscale moves on to explain his work on sentry guns. Sentry guns will now shoot anyone with a criminal flag, suspect or otherwise. Sentry guns will also start with smaller amounts of damage, and ramp up with time. Ideal tuning will be to where triage carriers will die at around 4 1/2 minutes. This way, if you want to use triage carriers in lowsec on gates you can, but you must commit to the cycle for a length of time before starting your reps, if you want to deactivate triage before the sentry guns kill you and jump out. CCP Greyscale also points out that another goal is to make it so that the first couple of hits won’t kill an interceptor immediately, enabling a quick tackle, and then a warp out.

Aleks remarks that this would be great for enabling more frigate use in lowsec piracy.
Aleks asks when all of these changes will be released, and when there will be dev blogs released for this information.
CCP Masterplan explains that this is where everything is at in the design process, that they’re looking forward to working more on this as the Inferno stuff dies down.
CCP Soundwave: “It is looking like a December release.”
Aleks and CCP Greyscale briefly discuss community response to these changes, Greyscale acknowledges that the changes to “suspect” flagging would upset some players, particularly canflippers.

I want to be clear that presentation of this section of the CSM minutes is very different from this following section, which is clearly a brain storm:

On the subject of sniping, Greyscale tossed out a high-level idea for a fix to sniping. He asked for CSM input on one such idea, an interdiction probe that would be launched a certain range before the bubble would deploy. In essence it would work as a drag-bubble to protect the sniping fleet, or at least give it ample time to react and reposition.
Elise was receptive to the idea and added that the biggest hindrance to sniping is the speed of ongrid probing.
Seleene and UAxDEATH agreed strongly.
CCP Soundwave chimed in with an idea of putting probes on grid and making them destructible. He argued that it would give small support a more pronounced role.
UAxDEATH was very receptive to the idea of giving support a stronger role.
Two step spitballed some ideas where probing would become less accurate with more results, and another idea where there was a probe-killing-probe.

The rage was further fanned on by certain CSM members who seemed to be all but directly confirming that it was already in development and would be hitting TQ in a form very like it was presented in the CSM minutes.   The problem got worse when many other CSM members refused to clarify that section of the minutes or actively defended it.  There were many suggestions by both the CSM and CCP to post on the forums – because obviously this is a good place to hold a discussion.  The only official answer was “It isn’t final until it’s on TQ”, a phrase that’s preceded a great many things that have in fact made their way to TQ.  It truly surprises me that the CSM feels that the community “jumped the shark” and was unreasonably angry given the presentation we were given about what was said.

However, after much discussion with various CSM members via blog post comments, forum posts, twitter, and Skype… I have to say that I’m pretty sure that the CSM minutes probably did not reflect the spirit of what was said at the CSM summit.  Both Hans Jagerblitzen and Seleene specifically said they’d not support any changes with ramifications so broadly destructive to PVP in low sec.  Hans even went back and watched the video and says that the context surrounding the fact it was brain storming was simply not put in the CSM minutes.

And as a final recap, here is a sample of the enormous list of problems with what was represented in the CSM minutes:

  • Gate camping is a hallmark of PVP in Eve because travel chokepoints are where you are going to find people.  This is true in high sec, low sec, null sec, and WH space.  Why should low sec suddenly become very different without core game mechanics changes that support that change across all of Eve?
  • Any sentry fire that was heavy enough to kill triage carriers at 4.5 minutes would be dealing somewhere between 35k and 150k DPS – obviously much more than any subcap can work around.  This means that the overall implication is that most non-gank subcap fights would also end up with everyone just getting blapped by the sentries.  Gate camping and ganks would still be possible via ninja camping and sniping, but real PVP fights would simply become untenable to have on a low sec gate.
  • The suggestion allows for gate camping with triple sensor boosted stilettos.  While it won’t materially change my own travel through low sec, this would make travel for non-flashy people much more dangerous – ostensibly something that we’re trying to avoid.
  • The suggestion does nothing for the core reasons why low sec is an underutilized area of space.  Provably, gate camps were never the problem – after all, we can look to null sec with it’s perma bubble camps to know better.  It’s must be a risk vs reward thing.

So obviously the suggestion itself is not fully thought out because of it’s ramifications for PVP in low sec, so let’s come up with something that sounds reasonable.  From everything I can gather, these are the primary motivations:

  • Get more people in to low sec
  • Allow different kinds of engagements on gates (eg, frigates)
  • Prevent perma camps
  • Prevent capital camps
  • Prevent orca camps?
  • Prevent blob camps?

For the sake of full disclosure, here’s what I personally think of each:

  • More people in low sec is great, but I don’t care so much about carebears coming to low sec.  They will never leave the relative safety of High and Null sec for the much more dangerous waters of Low sec – and that’s perfectly fine.  Give me your casual and small gang PVPers instead – all of them you can round up. 🙂
  • I’m kinda -1 to letting frigs engage under sentry fire.  On the one hand it’d be cool, but on the other it makes travel through low sec much more dangerous for carebears.
  • While I don’t tend to gate camp (it’s boring), I see absolutely no problem with perma camps.  I feel like places like Amamake and Rancer provide interesting geography and places of legend.  I see this as literally no different than the perpetual camps in PF- and M-O and other null sec entry systems.
  • I see absolutely no problem with capital camps.  I feel like low sec tends to small gang PVP and dropping a triage carrier or two is a pretty big signal there’s about to be an epic fight.  Well, it would be if you weren’t about to have PL drop a few hundred supercaps on you anyway.
  • Yeah, nerf the shit out of Orca/Carrier stowing under aggression.
  • I’m not a big fan of blob camping, but it’s allowed in every other area of space.  I see no reason why it shouldn’t be allowed in low sec too.

If the goal is to allow more types of PVP in low sec, I’d say that the first thing that should be done is just remove sentries entirely.  The key distinction between NPC null sec and Low sec would be the lack of bubbles, bombs, and certain supercapital features.  It’d mean that people no longer hesitated to pull the trigger on who would aggress because there would be no sentry fire to worry about.  There are a number of really good things about this approach – however it’d also kill the “heavier ship” fighting style that only blooms in low sec.

If the goal is to get more carebears into low sec, I think we’re looking entirely in the wrong direction.  The simple fact of the matter is that anyone that can stomach any risk at all is already in null sec – where the best rewards are.  There’s no reason for anyone to go to low sec for carebear rewards, ever.  So the first thing to do would be to provide that.  Then we should keep sentry guns or perhaps buff them a bit (with the added “benefit” of further encouraging the heavier ship doctrines I mentioned earlier).  Even if we neglected the fact that most established pirate corps have dozens of max skill scan prober alts, the instalocking frigs would simply be the doom of high sec carebears coming to low sec for PVE.

If the goal is to nerf Orca/Carrier stowing, then simply prevent stowing when you could not jump through a gate (eg, you are aggressed).  You could also transfer aggression to the Orca.

If the goal is to nerf capital camps and encourage frigate PVP under gate fire, I’d say the right answer is to turn sentry guns into missile batteries firing relatively slow missiles.  This means that you can scale damage by sig radius and speed, so capitals are getting hit by a dread while frigs and fast cruisers are getting tickled.  I hesitate to point it out, but it could also be used as a mechanism for ramping up the damage and would provide a visual indicator of how angry the sentry guns are becoming – warp out or die when this Ball O Rage hits you.

One thing I know for sure is that I am not a game designer and don’t really want to be one.  But, I will do everything in my power to help the game move in a direction that’s a bit more friendly to everyone without simply deleting my play style.

Filed under: Eve, Gaming

Calling the CSM to Account

So recently the CSM spent a great deal of time discussing their role and how important they are to CCP as a reality check/sounding board. I use the term “recently” a bit loosely, mind you – it’s been months since the CSM met with CCP, but only just now have the minutes come out. Inside those minutes, on pages 93-96 or so is some interesting information about CCP’s plans for gate guns in low sec:

Present: CCP Masterplan, CCP Greyscale, CCP Unifex, CCP Explorer, CCP Punkturis, CCP Tallest, CCP
Tuxford, Alekseyev Karrde (Lync), Hans Jagerblitzen (Lync)

CCP Greyscale moves on to explain his work on sentry guns. Sentry guns willnow shoot anyone with a criminal flag, suspect or otherwise. Sentry guns will also start with smaller amounts of damage, and ramp up with time. Ideal tuning will be to where triage carriers will die at around 4 1/2 minutes. This way, if you want to use triage carriers in lowsec on gates you can, but you must commit to the cycle for a length of time before starting your reps, if you want to deactivate triage before the sentry guns kill you and jump out. CCP Greyscale also points out that another goal is to make it so that the first couple of hits won’t kill an interceptor immediately, enabling a quick tackle, and then a warp out.

Aleks remarks that this would be great for enabling more frigate use in lowsec piracy. Aleks asks when all of these changes will be released, and when there will be dev blogs released for
this information.
CCP Masterplan explains that this is where everything is at in the design process, that they’re looking forward to working more on this as the Inferno stuff dies down.
CCP Soundwave: “It is looking like a December release.”
Aleks and CCP Greyscale briefly discuss community response to these changes, Greyscale acknowledges that the changes to “suspect” flagging would upset some players, particularly canflippers.

There are several overall implications here:

  • The first sentence describes this work as already being somewhere between in progress and finished.  Either way, these changes are expected to be complete and ship in December 2012.  CCP and the CSM have indicated that this is a miscommunication in the CSM minutes.  It was apparently not presented as it was presented in the CSM minutes – it was more of a brain storm session with much rabbit holing.
  • Sentry guns will now shoot people with a criminal flag (GCC, sec status below -5) as well as people who are flagged as a suspect (can flipped, minor crimes of aggression).
  • It will now be reasonable to run a 3x sensor boosted Stiletto on the gate for initial tackle with long range sniper tornados for killing the captured prey.  Solo travel through low sec will become much more dangerous.
  • The goal of killing triage carriers (2-3M EHP, 25k DPS tank) at 4.5 minutes means that the scaling on sentry fire will average much higher than 25k DPS at 4.5 minutes.  Some basic exploration of the matter says that the sentry guns will be dealing somewhere between 60K-125k DPS depending on when the exponential scale starts to invoke (at 2.5 minutes and 4 minutes respectively).  Pretty much no matter how you slice it, at 4.5 minutes into a fight we’re going to be seeing super tanked subcaps blapped off the field by sentry fire.
  • The net implication is that it’s a massive boost to gate camps and a massive nerf to actual fights on a gate.

The community is, of course, split over this:

  • Carebears are trumpeting about how pirates have systematically killed low sec through over hunting and gate camping.  Why, it’s about time that CCP did something to push pirates into getting real fights in belts like they did back in 2003!
  • Issler (CSM, ultra carebear who desires to eliminate PVP in Eve) is pointing out that gate camping has never been an allowed mechanic, which is why gate guns were first implemented.  They’re just fixing the bug that allowed pirates to gate camp in the first place.
  • Gate Camping pirates don’t seem to care or are excited that they’ll now be able to tackle with a 3x SeBo’d Stiletto and snipe with Tornados.
  • People who roam are crying bloody murder because virtually no fight of any size resolves within 4 minutes.
  • Very few people have noticed that pirates are going to be shot for simply waiting on the gate, regardless of aggression status. This is wrong.

In response to the pure ignorance displayed by CCP Greyscale and the CSM, I have offered to sacrifice my main character (-10 sec status, 90M SP, pure Minmatar PVP spec) for the purpose of forcing CCP Greyscale to play a pirate for the next 6 months.  I will furthermore fund the account for this endeavor so that he can keep skill training with his 3 free (presumably industrial) accounts.  While it would be ideal if I were to get the character back, I would understand if it weren’t possible.  I feel that offering to sacrifice my 90M SP main, the name my friends know me as, and $100 USD of my own money is the most sincere way that I can express myself on this matter: CCP Greyscale cannot possibly have played the game as a pirate (or PVPer in general!) in recent memory.  I feel that his education is absolutely critical.

Now, while I am disappointed with CCP on the matter, I am even more disappointed with the CSM.  So it is at this point that I demand an accounting of the CSM.  They claim their job is to protect CCP from making boneheaded mistakes, yet not even one of them objected to these sentry changes.  Not even one of them had the foresight to understand the catastrophe that is going to happen to low sec.  Not even one of them realized that this is effectively deleting the entire pirate profession from the game.

So what gives, guys?  Why didn’t you object to deleting PVP real PVP from low sec?


Clarification from Masterplan [1][2]has stated that pirates will NOT be blapped by gate guns without first having done “bad things”.  Furthermore, the suspect flag will probably not cause blapping.

Filed under: Eve, Gaming, , , ,