Steam Dev Days – Data to Drive Decision-Making

Steam Dev Days – Data to Drive Decision-Making by Mike Ambinder, Valve

How and Why Valve uses data to drive the choices they make

Mike is an experimental psychologist – takes what he knows about human behavior and applies it to game design.

Data to Drive Decision-Making

  • Decision-Making at Valve
  • Introduction to experimental design
  • Data collection/analysis infrastructure
  • Examples: LD4, DOTA2, CS:GO

Decision-Making at Valve

  • No formal management structure
  • Decision-making is meritocracy
  • All data is available for every employee
  • We just want to make the best decisions possible
  • We don’t want it to rely on ‘instinct’ -> it is fallible
    No centralized command hierarchy – as such decision-making is a meritocracy. [Huh? Who, what, how without linkage. What about regression to the mean? How is “merit” determined? The more times I hear this, it seems shockingly political. “Spending lots of time making good decisions” implies to me that there is some rubicon to evaluate them. How is that not a political process?] All data is made available with the exception of employee compensation. By instinct he really means let our biases run amok.


  • Explicit
  • Data-driven
  • Theory-driven
  • Measurable Outcomes
  • Iterative


  • What problem are you trying to solve?
  • Define terminology/constructs/problem space
  • Ask the ‘second’ question
  • Force yourself to be specific
  • Force yourself to be precise
    ‘Second’ question -> “What do you mean by that?” It’s a technique to dig into something to make sure comprehension happens, that you’re specific and precise, that there’s consistent logic and supporting data.


  • What do we know about the problem?
  • What do we need to know before we decide?
  • What do we still not know after we decide?
    Need to know what you know – and what don’t. Being honest with yourself about that is important.


  • What does the data mean?
    ** Is it consistent with expectations?
    ** Is it reliable?
  • Model derived from prior experience/analysis
  • Coherent narrative
  • Prove a hypothesis right (or wrong)
  • Want result AND explanation
    Behavior during a Steam Sale is different than not, so make sure you consider that. Have sufficient confidence in your data using statistical analysis. You want to have some “intuition” about why something happened. A narrative. [Odd choice of words that…] Even if you don’t know for sure have at least a hypothesis for what’s going on – and then set out to prove it correct or wrong. The goal is to make smarter decision in the future.

Measurable Outcomes

  • Define ‘Success’
  • How will we know we made the right choice?
  • Know the ‘outcome’ of your decision
    Know what success is for every decision you make. If your decision is loosely tied to customer actions – how do you know it was a good one? Measure the outcome of your choices.


  • Gather Data
  • Analyze Data
  • Formulate Hypothesis
    Data from one game informs decisions in other games. “TF2 is a test-bed for DOTA2, and vice-versa.”

Introduction to Experimental Design

  • If it can be destroyed by the truth, it deserves to be destroyed by the truth.” – Carl Sagan
    We all want to be right all the time. Valve would rather be accurate than right. They want estimations of how reality is to match what reality actually is.

The Scientific Method Cycle [YAY!]

  • Theory – use the theory to make a prediction
  • Prediction – design an experiment to test the prediction
  • Experiment – perform the experiment
  • Observation – create or modify the theory

Experimental Design

  • Observational
    ** Retrospective vs. Prospective
    ** Correlational not causal
  • Experiment
    ** Control Condition and Experimental Condition
    ** Account for confounding variables
    ** Measure variables of interest
    Try to eliminate external influences.

Experimental Design (Part 2)

  • What have we learned?
  • What biases are present?
  • How are future experiments informed?
  • What other hypotheses need to be ruled out?
  • What should we do next?

Data Collection/Analysis Infrastructure – Valve Data Collection

  • Record lots and lots (and lots) of user behavior
  • If we’re not recording it, we’ll start recording it
  • Define questions first, then schema
  • Collection -> Analysis -> Communication
    Always willing to spend engineering time to get the data to answer the questions they have. They never regret that. It doesn’t mean they’re always right – but they’re always smarter. Once you have the data you need to have an idea of how you’re going to share it.

Data Collection – Games

  • OGS – Operational Game Stats (?)
  • Platform for recording gameplay metrics
  • Kills, Deaths, Hero Selection, In-Game Purchases, matchmaking wait times, Bullet trajectories, Friends in Party, Low-Priority Penalties, etc.
    They records “everything”.

Data Collection – Games (2)

  • Organizational schemas defined for each game
  • Data sent at relevant intervals
  • Daily, Monthly, Lifetime Rollups, Views, Aggregations
    [These data collection examples are Valve games only. There’s no Steam provisioning for this sort of metrics collection. I wager there are partners who’d want that.]


  • Disseminate the data using Tableau
  • Examples:
    ** Account First Purchase
    ** Chinese Users Performance
    ** DOTA Heroes
    ** DOTA Item Balance
    ** DOTA Matches
    ** DOTA Geographic Purchases
    ** DOTA Item Purchases / Drops
    ** DOTA Sales by Currency
    ** DOTA Weekly
    ** DOTA Performance
    [Charts are really hard to read, so no scale or value data readable. Probably available elsewhere if required.] Have 200 separate workbooks, about 800 pieces of analysis.

Data Collection – Steam

  • Steam Database – Raw data
  • SteamStats Database – Analysis/Summary of raw data
  • Record all relevant data about Steam user behavior
    [Screenshot of SteamWorks Product Data screen at 24:19] He made an interesting comment about if ARPU or ARPUU are good metrics to use. [Seemed to downplay their significance. Not surprising given the Trade System examples and free user monetization strategies that they use.]

Valve’s Game Design Process

  • Goal is a game that makes customers happy =>
  • Game designs are hypotheses =>
  • Playtests are experiments =>
  • Evaluate designs based off play test results =>
  • Repeat from start =>
    We are very poor proxies for their customers. They don’t know if something actually works until they put it in front of people who are not them.

Playtest Methodologies

  • Traditional:
    ** Direct Observation
    ** Verbal Reports
    ** Q&A’s
  • Technical:
    ** Stat Collection/Data Analysis
    ** Design Experiments
    ** Surveys
    ** Physiological Measurements (Heart Rate, etc.)

Example – Left 4 Dead – Enabling Cooperation

  • Coop Game where competing gets you killed
  • Initial playtest were not as enjoyable as hoped
  • Initial playtests were not as cooperative as hoped
    ** Players letting their teammates die
    ** Ignoring cries for help

Enabling Cooperation

  • Explicit: Players letting teammates die
  • Data-Driven: Surveys, Q&As, high death rates
  • Theory-Driven: Lack awareness of teammate
  • Measurements: Survey, Q&As, death rates
  • Hypothesis: Give better visual cures to teammate location
    Improving the visual queues caused deaths to go down by ~40%. [Duh. The previous version was clearly inadequate.]


  • Survey rating of enjoyment/cooperation increased
  • Anecdotal responses decreased
  • Deaths decreased
  • Iterative: Where else can visual cues aid gameplay?

Example – DOTA 2 – Improve Player Communication

  • Explicit: Reduce negative communication
  • Data-Driven: Chat, reports, forums, emails, quitting
  • Theory-Driven: No feedback loop to punish negativity
  • Measurements: Chat, reports, ban rates, recidivism
  • Iterative: Will this work in TF2? Do these systems scale?
  • Hypothesis: Automating communication bans will reduce negativity in-game.
    They had data which suggested that they had a problem. The (early-on) only significant predictor for why a person would quit DOTA was being in a game where a player had been reported for abusive behavior. Rewarding positive behavior is a different axis. The way it works (38:09) is the player gets a report player dialog which categories the report (i.e. Communication Abuse) with a free-text more information box. They also get a Thank You dialog which specifically tells the player that Valve has taken action against another player and that they have another (note singular) report to use. Players have a weekly quota of reports. [Both of those are really interesting feedback loops. I’m not coming up with any other games which do this? Every game I can think of specifically does the opposite of this.] They take away the other players ability to chat scaling from a day to a week depending on severity and frequency of bans.


  • 35% fewer negative words used in chat
  • 32% fewer communication reports
  • 1% of active player base is currently banned
  • 61% of banned players only receive one ban
    [Missing is what this has done to quit rates.] They balanced the word list to stay around the 1% mark to avoid overdoing the banning. [Not stated is how many reports for a particular player are required to automatically ban a player.]

Example: CS:GO – Weapon Balance

  • Explicit: M4A4 usage is high; few choices in late-game
  • Data-driven: Purchase rates
  • Theory-driven: Greater tactical choice => Player retention
  • Measurements: Purchase rates, playtime, efficacy
  • Iterative: Inform future design choices
  • Hypothesis: Creating a balanced alternative weapon will increase player choice and playtime
    The M4A4 was too popular – 80% of players. Could be good, but wasn’t sure. They introduced the silenced M4A1 which split evenly with the M4A4 purchasers.


  • ~50/50 split between new and old favorites
  • Increase in playtime
    ** Conflated with other updates
    ** Difficult to isolate
  • Open question as to whether or not increased weapon variability increases player retention

Where Can You Begin?

  • Start asking questions
  • Gather data – any data
    ** Playtests
    ** Gameplay metrics
    ** Steamstats
    ** Forum posts/emails/Reddit
  • Tell Valve what data you’d like them to provide

Contact Info

  • Mike Ambinder
  • mikea AT

Question: How often do you get to isolate a single change?
We play as much as we can as often as we can. Twice a week, twenty people, for longer than a year for L4D. It’s going to be messy sometimes. You need to be aware that the data you have isn’t representative of the population at large.

Question: Data-driven approach to avoid mis-steps.
We make mistakes all the time. The way the company is designed makes that ok. They did not realize the customers had an expect ion. Now they have more informed policies about holiday events in the future.


Steam Dev Days – Embracing User-Generated Content

Embracing User-Generated Content by Tom Bui, Valve


  • Why User Generated Content (UGC) is important
  • Examples from the Steam Workshop
  • How to get started
  • Rewarding your content creators

What is UGC?

Content created by the community for the purpose of personalize or adding value to your product. [Duh.] Two axis for characterizing UGC:
* Aesthetics vs.Behavior
* Parameterization vs. Creation

UGC is a service

  • Provides ongoing value to customers
  • Exposes new ways to play your game
  • Gives customers a voice
    Supported by both you and your community. A vision of your game not bounded by just your resources.

You need UGC

  • The community will make your game better
  • Beat the competition
  • Customers will experiment
  • See what works
  • Change direction if necessary
    Games that embrace UGC will do better than ones which don’t.

Example – DayZ

  • Started as a mod of ARMA 2
  • ARMA 2 sales skyrocketed
  • Officially became a standalone game

Any game can benefit

  • Multiplayer & Single-player
  • Big & Small
  • All Genres
    Examples: TF2, Skyrim, DOTA2, Don’t Starve, Dungeon Defenders, Prison Architect, Drunken Robot, Duke Nukem 3D. If the customer’s love your game – they’re going to mod it.

Example #1 – TF2

* In-game cosmetic items

Order of operations:
1. Content creators upload their files to Steam Workshop
2. Community reviews & votes.
3. TF2 Dev team vets.
4. Made available by purchasing or by playing the game.
One of the main reasons Valve built the Steam Workshop was so the community could review the items before Valve did.


  • 25% revenue share: Content creators receive 25% of the revenue from the direct sales of an item.
  • 7,850 items in Steam Workshop.
  • 514 items in TF2.
    The compensation has improved quality dramatically.

Example #2 – Skyrim

* New Weapons
* Custom Quests
* Gameplay Modifications
* Texture Updates
Centralized mod distribution and made it easy. [Man, no kidding.]


Holy Camoly!

Are they counting Bethesda DLC purchasers in that number? Seems really high. Shockingly so.]

Example #3 – Don’t Starve

* Language Packs & Tutorials
* Unique Characters
* New Items
* Gameplay Modifications
* UI Mods
Particularly good support for game personalization.

Example #4 – Counter-Strike:Global Offensive

* Maps
* In-game Items.
Solved centralized place for map distribution problem. The Steam Workshop integration is on the Game Server side with guaranteed auto-updating.

Items follow the same model as TF2 with community moderation. Over 20k skins.

Example #5 – Killing Floor

* Maps
* Characters
* Weapons
Used a top-rated content strategy but bundled into specific DLC.

Example #6 – Red Orchestra 2

* Custom maps
* Mods
Used a contest with a $35,000 prize pool – twice.

Example #7 – Portal 2

* Custom maps
Super easy map editor has resulted in over 381,000 maps. Over 3.5 years of non-stop gameplay. Interesting many top rated maps were not created by the easy-to-use tool. [Hmm…]

Example #8 – Source Filmmaker

* Maps
* Models
* Animations
* Effects
Makes posters, movies, and comics. Used to promote other UGC

Example #9 – Garry’s Mod

* Unique Characters
* New Items
* Gameplay Modifications
* Language Packs
* UI Mods
It’s seemly sole purchase is to create more UGC to share with other players.

72 products integrated with Steam Workshop as of 2/11/14

Getting Started

You should start right now

Steam Workshop

  • Searchable, centralized repository
  • Hosting, infrastructure & management
  • Rating, favoriting, sharing, etc.
  • Continual support & new features
    [The hosting thing is a big deal. Interesting, the revenue split is much more akin to Amazon’s Kindle revenue share rather than Apple’s App Store one. There’s a HUGE difference between 25% & 70%. Yet aren’t both “stores” doing basically the same thing? Is the API add-on & Steam client that much of a value add? (Now granted search in the App Store – Oy vey.)

Start with what you have

  • Start small
  • Keep it simple
  • Iterate
    Focus on one type of UGC and expand from there. Swallow your pride and get started.

Don’t limit opportunities

  • Dynamic range > ease of use
    Buttressed by the fact that the best Portal maps are done in the more complex and rough Hammer editor instead of the easy-to-use one. More power in the hands of the user is the right choice.
  • Allow free form submissions
    Allow users to tell you what they want in the game
  • Embrace external tools
    Examples: Java MD3 Model Viewer. SimPE Editor (Sims Package Editor).

Share Your Resources

  • Assets
  • Source Code for Tools
  • Data
  • Documentation
    [Can this be “safely” done in the App Store? They’re likely to do it anyway – but that just makes it faster.]

No Documentation?

  • Set up a wiki or point to Steam Guides
  • Let creators help you
    Centralize and support it as much as possible. [I like this.]

Iterate and Improve – Incorporate feedback

Learn from your customers

  • What are they trying to do?
  • How do they want to play your game?
  • Which tools need the most work?

Iterate Deliberately

  • Improve your tools where necessary
  • Or, support community that is doing it for you
  • Expand to new types of content
    The opportunity cost of letting the users give you UGC early is worth it.

Feature mods in-game

Make the lives of your UGC consumers easier. Ask users to vote and use that to figure out what the best content is. Make it all seem like one seamless experience.

Rewarding Creators

Encouraging top quality content

Build toward money

  • Financial compensation for creators is critical to ongoing quality content.
    [How can you do with code changes to tools?]

Run contents

  • Offer prize money to top-rated content
  • Ship winning content to all customers
    Works well on an intermittent basis

Release as DLC

  • Bundle up some top content
  • Polish/Optimize and sell as DLC
  • Pay the creators a share of the sales or flat fee
    [This would work in the App Store as an IAP. Is there some reason why this isn’t already happening because of the ToS?]

In-game sales

  • If you have an in-game economy
  • Accept items created by users
  • Sell them in-game
  • Pay the creators for a share of sales
    Steam Workshop does the creator pay-outs for you. Customers tell the developer AND the creators which one they like best with their dollars.

Service Providers

  • Tool vendors support TF2 item creators
  • Communities support item creators
  • It’s in everyone’s interest to support those vendors and communities
    [Blizzard is falling down on this. Why?] 5% of Steams share of the revenue go to tool vendors (i.e. Service Providers.) Examples: Polycount, Handplane, Blender. [O_o. Did not know… This further cements Valve as the games to build your portfolio in.]

Just the Start of UGC

Everything that users create that adds value to your product [sic]

UGS is everything


  • UGC makes your game more valuable
  • Steam Workshop can help
  • Start now and grow your community
  • Think outside the box

Question: How do you protect yourself from theft?
Valve doesn’t worry about that. Source code itself isn’t the value, it’s the developers. Used HL2’s source code release as an example. The people provide the value. Your execution.

Question: Do “unofficial” servers which bypass item rarity restrictions damage value?
No. The value is derived from the community on the “official” servers. Aka Monty Haul syndrome.

Question: Different Steam Workshop integrations – one with revenue share and one without.
Documentation isn’t done yet.

Question: How do you verify copyright for UGC?
Valve has a DCMA process. The community does a fairly good job of moderating the content themselves via down-voting and reporting.

Question: How do you deal with multiple play locations – not just on Steam?
Skyrim does already support this. There’s no Steam restriction on play here. [Non-answer.]

Question: What do you do to prevent backward compatibility breaks of UGC?
Some “partners” use beta branches of their games. They’ve invited those top moders to test them. [Pretty weak answer really. A more complete answer is you have to design your game to not do BC breaks unless absolutely necessary. Welcome to legacy software support – aka software as a service – aka online games. But that’s not something “game” developers want to hear and he was in partial sales mode.]

Question: How do you get internal artists to buy into UGC more?
[Got him to laugh, so clearly this has arisen.] They had this exactly problem on DOTA2. The community proved that it could do it. As artists, they loved that, so they took more of a Art Director role instead.

Question: Aren’t you giving up control over your art aesthetic?
They did have that concern for TF2 and DOTA2. That’s why they have a curated model for the Workshop. They maintain veto authority. [Duh.] It is giving up control, but that’s OK. [It’s right on that point which Blizzard has the most resistance, even though they support mods. They do not incorporate UGC into the shipping product. Allowing users to add it into their game via StarCraft2 Arcade is really as close as it comes. Even then, that only just got “turned” on for everyone just recently. Curious to see how that works out long term. Also note, no shared financial renumeration at this time.]

Question: How do you handle the security aspects from a malware perspective?
Give nodders as much control as possible but sandbox them in such a way that they can’t affect other players. Example: Lua can be sandboxed to project the base-OS. Another approach is up-vote/down-vote as another (suboptimal) approach.

Question: Long term concern, how does this scale to 1,000 games with a more diffuse contributor/creator-base? DOTA2/TF2 has a bit of a gold rush at the moment.
Not concerned because there’s a lot of great content creators out there. Only tapping a tiny portion of them. Follow up question, using Pinball Construction Set as an example where nodders got burned out. [Steam, and other centralized locations on the Internet, address that issue. It’s really a non-sequitur to bring up a game from before the Internet, especially in todays market.] They gained more artists as they add more Workshop games. They’re not seeing (many) artists migrate from one game to the next, because they have a favorite game. Professionals will optimize for their own personal revenue, but there are many who do it because they love it.

Question: How do you monetize maps without segmenting the community?
CS:GO took community maps and put them on official servers which cost money to play on. They gave 100% of the proceeds to the map creators.

Steam Dev Days – In-Game Economies in Team Fortress 2 and Dota 2

In-Game Economies in TF2 and DOTA2

Focus on making your product better.

Use Micro-transactions and economic systems to improve the customer experience. Just using them to extract value will fail.

Part 1 – Lessons Learned & Recommendations

Recommendation #1 – Focus on Persistent Customer Value.

They have to be able pass the “regret” test. If it comes at the cost of customer happiness, don’t do it – even it means that someone might not become your customer. You’ll get them later once you figure out how…

Regret Avoidance Tools

  • Communicate clearly up front (Store front & checkout)
  • Maintain that value over time (Game Design – i.e. Trade Systems)
  • Metrics tracking of customer usage (Back-end systems)

Regret Generation Tools

  • Artificial barriers in the game. Clearly there to extract revenue and provide no service to the customer – aka every appointment-based F2P game ever.
  • Virtual Currencies which obfuscate value.

Recommendation #2 – Positive Externalities

More players spending more makes for a more positive gaming experience. Systems which cause you to have more fun, because someone else spent money. Improve the quality of the game for everyone. Used to evaluate existing designs and as a starting point for new designs.

Recommendation #3 – Make Everything Tradable

Trading makes every item and system in existence more valuable to more people. Every system which interacts with trade becomes more valuable as a result. Trade can become a positive thing for everyone involved – because two customers are interacting together.

Recommendation #4 – Distribute Value Randomly

Random distribution is another tool to generate player engagement and concrete value. Other games use “static” distribution systems with fixed reward schedules based on parameters like time played. That limits what you can offer as a designer. Via random, item values can be dramatically different.

Recommendation #5 – Let Users Make Value for Each Other

Enlist the Internet to fight to make your product even better. Some are amazingly good – use the players of your games. [Valve does this really well…] If you don’t allow them to do so, they’re going to improve someone else’s. [From a hiring perspective, a game company should aspire to be the “reference” standard for portfolio pieces that students produce in school. Valve is well on their way here…]

Reward people financially relative to the value that they contribute and/or generate.

List of possible targets:

  • Comics
  • Movies
  • Animation
  • Tutorials
  • Community
  • Leagues
  • Crafting
  • Trading
  • Moonbase (?)
  • Modeling
  • Gifting
  • Maps
  • Levels
  • Mesh Content
  • Organize groups of like-minded people

Part 2 – Individual Case Studies

Case Study #1 – TF2 Gifts Data (as of 2/1/14)

  • Sent Gift: 1,067,399 accounts
  • Received Gift: 1,841,051 accounts [ratio is closer to 1:1 than I would have expected…]
  • Big gift bundle: 10th highest lifetime revenue generator
  • Current leaders: 12,355 gifts given
  • Given over 1,000 gifts: Over 140 accounts


  • Impetuous – what is something that you can buy which causes me to start celebrating when you make that purchase? What is a system which can generate positive externalities (Recommendation #2)
  • One time consumable.
  • Everyone on the server gets value from your action.
  • You get no direct value.
  • You get social recognition.

Case Study #2 – TF2 Crates & Keys


  • Impetuous – Let players opt-in to random distribution system. This was an experiment to find out if players where interested in spending their money to do this.
  • Most negative feedback around store launch
  • Perception problems:
    ** Capitalize on poor judgement
    ** Maximize revenue extraction
    ** The community ascribed intentions to their actions. Opposite of Recommendation #1.

Follow-on Actions Taken

  • Remove ability to hard-lose real currency on open [What this means is that the items you could get from the crate were worth less than the what the key cost to open. This provoked more negative emotion than positive because the loss generates more than gain. Basic Psychology 101 error… Established a clear rule that the items had to cost more than the key – eliminated the possibility for loss to occur (because trading exists.)]
  • Put users in a situation where they had random chances – not gambling [in the Vegas sense.]
  • Add variety: more types, more contents
  • Add value: common items from crates add rarity-agnostic services. (i.e. crate guns provide gameplay tracking statistics that regular ones don’t.)
  • Participation is higher. Increased community engagement. If they don’t ship crates fast enough, users get angry.
  • “Better is not “done”

Conversation Rate

  • 13% Purchased
  • 15% Opened
  • 75% Owned Item
    This isn’t that useful and potentially damaging. 3 out of 4 people are generating demand and consuming content which comes out of the crate system. The trade system allows Value to effectively monetize free players, because they generate demand for goods which other’s purchase – even if they never spend any money. Everyone along that chain is happier as a result of completing those transactions.

Random Distribution and Trading

Use valuation differences to generate revenue for you and your players, all the while making players happier. Trade combines with random distribution systems in a way which is beneficial for everyone. “Incredibly positive.”

[OK SERIOUSLY.] “Users are potentially trading for keys, because they don’t have credit cards or Valve has poor payment services.” [NOT FOR THE LACK OF TRYING. Geese.]

[Need to analyze the DOTA2 value chain.]

Case Study #3 – Community Content

  • Over 90% of item content from community:
    ** Models, maps, and much more
    ** Marketing (Users make animated videos, amazing art, etc.)
    ** Evaluation (community does this through the Steam Workshop)

Contributor Payout

  • 2010: $590,900 for 106 items by 63 creators
  • 2013: $10,215,796 for 2,349 items by 661 creators
    The community is evaluating the user created content and “voting” with their own money what that content is worth.

Community Content

  • Entirely community-made:
    ** Marketing
    ** Movie
    ** Comic
    ** Organization
  • Then-highest single day revenue in TF history
    Community did their own complete patch of TF2. All of those people collected revenue based on their work.

Case Study #4 – Trade – Item Visibility/Demand

  • “Trade” is a basic idea:
    ** More consumers add value
    ** More uses add value
  • Increase visibility/demand:
    ** Trade as a Steam feature
    ** Community Market
    ** Still growing
  • Partner games benefit in the same way
    If you have an item you can link to it from everywhere. You can trade across games. Steam Marketplace allows users to trade items for Steam wallet funds. The value of ever item in the economy as more games join the economy – there’s no fall-off and it’s reciprocal.
    Daily revenue of some games have gone up by over 50%

Case Study #5 – DOTA2 Leagues

  • Funds split with league organizers, used for prize pool, production, etc.
  • Average 1:8 league viewers traded for ticket, up to 1:4
    Leagues work similar to pay-per-view. Watch the games live. Historical access. Follow a particular player’s mouse movement. 100% spectator.
    Because these tickets were tradable – these leagues all have bigger audiences. Monetizing free players – while making everyone happier.

Case Study #6 – DOTA2 Battle Boosters

  • Goal: “I celebrate when the guy next to me buys one” – Same place as the TF2 Gifts design.
  • Rise of positive, lowering of negative comments at round start. Sea-change level of reaction.
  • Increate the rate at which all players get random drops – even if an opponent buys one.
  • Another free player monetization strategy because the free players are some of those making the positive comments.
  • Iterated to clarify value proposition – because they want more of this.

Case Study #7 – DOTA2 – The Interactive Compendium

  • Positive externalities:
    ** Stretch goals (Kickstarter-like thing.)
    ** Battle Booster (Which effected you, even if you didn’t buy the book.)
    ** Prize pool
  • Results:
    ** 484,768 Sold
    ** ~$1.2m added to prize pool (Single largest eSports event ever.)
    ** Community rallying cry
    DOTA2 tournament that Valve runs. Interactive program with historical game. Fantasy “football” like mini-game. Vote for All Stars. [Think every sports fan participatory thing ever.] Doesn’t fit into any existing model of Micro-transactions. It was a surprising new thing.

Recommendation #6 – Explore!

There are so many ideas out there which don’t fit into existing models. This space is relatively new. Allow everyone to be able to make you’re games more interesting – artists, programmers, experienced players, EVERYONE.

Success Example: TF2 Today

  • 17m accounts owning items
  • 500m total items
  • 4B actions performed on items
  • 5x monthly players (600k -> 3M)
  • 4x daily free items dropped
  • 9x daily items sold
  • 6x daily revenue
    Game continues to grow – which means that the decisions they’re making are keeping their customers happy.

Everyone can win

Value rejects the premise that micro transaction systems must come at a cost of customer happiness.

[It is possible for everyone to make money and come away happy. You just need to think it through – have that as a goal – and it can be done. That is the net best thing for your, and everyone else’s, game.]

Follow Up

Kyle Davis: robin at


Holacracy Main Page

I haven’t read this in enough detail yet, but this feels A LOT like the Value organizational structure with one exception. There’s the concept of Leads (using a different word: “lead links”) around with there’s a least some formalization of structure. I haven’t teased out what the difference is between a Lead and a Manager is (with their definitions.)

Queue the Star Wars quote, but what about the bureaucracy? How will the governors maintain control?

The one thing I haven’t figure out is, aside from the project tasking & definition, etc. that “managers” or “leads” or (I imagine) “lead links” do, there’s the HR heavy lifting. It’s just a fact of life in a company larger than 5, let alone one as large as Zappos (1,500)…

Need to read more about this…

Zappos says goodbye to bosses

The one thing makes me really interested is that Zappos did a trial with about 10% of their employees (aka roughly the size of Value) and it went well so they’re rolling it out company-wide…

Valve Handbook for New Employees Nobody has…

Valve Handbook for New Employees

“Nobody has ever been fired at Valve for making a mistake.”

I love the sentiment. I hate firing people. The rest of the section of their philosophy on Risk reads really well and, in my experience, works really well.

I’m just not sure that is a good thing.

I suppose the issue is a literal definition of the word mistake. It’s one thing to get a game feature sufficiently wrong (defined as more people hate it than like it) that it could be considered a mistake. There’s so much subjectivity in game design AND the consequences for failure are not life threatening, so risk should be embraced, encouraged, and rewarded.

The problem is the word AND, along with the definition of the word mistake. Valve has had two security incidents which would make it into the category of life threatening. First, the disclosure of the HL2 source could be perceived as such, at least in term of the survival of Value. The leak of Steam credit card data, however, jeopardized the survival of Value AND (potentially) caused serious damage to their customers.

For the opening sentence to be Truth, the following must be true:

First, they’re using a much more expansive definition of the word mistake than I do. Mistakes are spectrum which also include fraud, negligence, and incompetence, not just “getting it wrong.”

Second, they might not even know who’s directly responsible for the conditions which lead to the breach in the first place. (Disclaimer: I don’t work at Valve, so I clearly don’t have first hand knowledge.) Every security breach scenario I’ve investigated had, as one of it’s root causes, people making errors which individually wouldn’t have caused the security breach, but in aggregate made one possible.

Wouldn’t a shifting self-selecting workforce create an environment absent ownership and stewardship?

How could that not create exactly the sort of seams through which hackers penetrate systems?

What happens when you interject actual human beings into this system, which all of their messy characteristics?

Would you want to develop Flight software this way? Put another way, would you want to trust your life to this software?

For me the answer is no. I just don’t like it. I suppose the answer is some sort of a hybrid for the smallest amount of critical functionality possible. Inside of that critical functionality bubble, something more conventional could exist, applying the well understood principals of mission critical software development, with a semi-permiable membrane through which people and functionality could pass as needed. The difference here is someone who be responsible for ensuring only “safe” things transited that barrier using APIs, code sharing, data passing, and system architecture. Outside of the bubble, the more amorphous and squishy team model could operate just fine.

“If you find yourself in a group or project that you feel isn’t meeting these goals, be an agent of change.”

Best sentence in the book. However the sentence before it:

“This handbook describes the goals we believe in.”

…closes the door on doing the hybrid-ish solution. They make it abundantly clear that they’re not willing to consider the hybrid solution proposed above.

I love the idea of taking risks. I just don’t understand how you get to the level of correctness required, when you never fire anyone and have amorphous accountability. My gut says that the peer review mechanism is supposed to weed the bad apples out.

“Peer reviews are done in order to give each other useful feedback on how to best grow as individual contributors.”

Nice thoughts. (I would have reworded those paragraphs to emphasis the positive nature of feedback, not just criticism.) Here’s the thing, if this is the only way you cause people to shift off of projects they’re not succeeding at, or out of Valve completely, as required, then there’s a Political problem. Political as in not Aristotle politics, but instead that messy partisan kind. Because after all, aren’t peer reviews provided by your peers? And in just about any group of people, how do you avoid the popularity contest issue?

They address some of this in the hiring section:

” It’d be tough for us to capture because we feel like we’re constantly learning really important things about how we hire people. In the mean- time, here are some questions we always ask ourselves when evaluating candidates:
• Would I want this person to be my boss?
• Would I learn a significant amount from him or her?
• What if this person went to work for our competition? Across the board, we value highly collaborative people.”

The precise questions used in their feedback process aren’t disclosed, but I get the feeling that they’re asking them again on a periodic basis. That’s key to ensuring that you haven’t made a hiring mistake. The use of the word “boss” is highly ironic, seeing as their definition doesn’t match the dictionary definition…

“Hiring is fundamentally the same across all disciplines.”

Awesome. Buttressed by what I’ve recently read about Pixar.

“We’re looking for people stronger than ourselves.”

Awesome. Made even more so by the fact that they justify and defend that statement.