1 Leveraging Human-ai Collaboration↩︎

AI and humans are frequent collaborators: in writing [1], making decisions [2], and creating artwork [3]. The most fruitful collaborations are those in which humans and computers have complementary skills, such as ai analyzing medical imaging to identify anomalies and doctors interpreting these findings. We posit that the board game Diplomacy is an apt testbed for studying this type of collaboration. [4] study cicero [5], the best Diplomacy-playing ai capable of communicating in natural language, and show that while the state-of-the-art ais have near-optimal move strategy, human players remain better at communication.

We introduce Personalized Help for Optimizing Low-Skilled Users’ Strategy (pholus),² a natural language agent that provides both moves and messages generated by cicero as advice to Diplomacy players in real-time. The core distinction between them is that cicero is a game-playing agent whereas pholus is an advisor that does not actively participate in the game. Players’ moves and message history influence pholus’s advice.

We run a user study and collect a dataset with twelve games, \(1{,}070\) player turns, and \(117\) playing hours. pholus enables novices—who barely know the rules of Diplomacy—to compete with experts (Figure 2). But this does not just mean the novices blindly follow the advice. First, they use pholus‘s strategic insights to inform their communication strategies with other players. Second, pholus helps experienced players, although they are less inclined to take the advice than the novices. Overall, both advice types from pholus improve players’ game outcomes (Section 3.1). Our research enables human-ai collaboration and offers valuable insights into the potential of using ai to enhance human learning experiences.

On a broader scale, our study explores the potential for ais like pholus to enhance learning in unfamiliar environments. ai agents surpass traditional rule-based methods by offering more flexible and personalized learning experiences. Integrating tailored guidance into human intelligence, these systems provide unique learning experiences for inexperienced individuals. Future research directions in human-ai collaboration include generating advice based on high-level intentions and goals, reducing over-reliance on skilled ais, and facilitating learning processes.

2 Diplomacy as a Cooperative Testbed↩︎

Diplomacy is a seven-player, turn-based board game. The goal is to obtain more than half of the board’s possible points.³ Critically, turns are simultaneous, with moves written in secret by players and then revealed. This means that players must communicate to collaborate effectively.

2.1 Experiment Setup↩︎

We recruit Diplomacy players online. For experienced players, we advertise in the Diplomacy community (specifically players active on webDiplomacy and Backstabbr, as well as in-person tournament attendees). To find novice players, we contact board game enthusiasts in university clubs. A novice player is someone who has no prior Diplomacy experience and is unfamiliar with its rules.

We modify a game engine and interface [7] and maintain the same game format used by [4]. Each game involves two to five human players. Games last about three hours, with each turn taking ten minutes.

As illustrated in Figure 1, pholus passively observes the game. If cicero is an active participant, it would have submitted moves and sent messages based on the game state and its message history. Instead, pholus presents these moves and messages as advice to players. Each time the player sends a message, pholus recomputes advice given the new context and presents it to the user. Every turn, we randomly assign each player to one of the following settings:
1) No advice: pholus does not offer any information, meaning the player receives no assistance from pholus.
2) Message advice: pholus suggests to whom a player should send a message and what the message content could be.
3) Move advice: pholus recommends a set of moves (or unit orders) to the player.
4) Message and move advice: Combines the previous two types.

In total, we collect data from twelve games involving fourty-one players. This includes over \(3{,}600\) entries of move advice and \(4{,}300\) pieces of message advice (Table 1).

2.2 Evaluation Metrics↩︎

To assess the effectiveness of pholus’s advice, we consider the net gain or loss of points in each turn as the effect of advice. We train a linear regression model with regularization to examine the advice’s effectiveness. The model includes features such as which of the seven Great Powers is assigned to the player, the number of turns that have passed, the player’s type (novice or veteran), and the advice setting. We encode the Power, player type, and advice setting as one-hot vectors.

To evaluate players’ reliance on pholus, we use both qualitative and quantitative methods. In addition to computing move advice acceptance frequency, we also measure agreement and equivalence between the move suggested by pholus and a player’s moves. Agreement is the proportion of moves that appeared in both the players’ move set and pholus’s advice set in a given turn. The sets are equivalent if they overlap entirely. Formally, we define move agreement \(\mathcal{A}\) in turn \(i\) as \(\mathcal{A}_{x_i, y_i} = |x_i \cap y_i| / |x_i|\) and equivalence \(\mathcal{E}\) as \(\mathcal{E}_{x_i, y_i} = 1\) if \(x_i = y_i\) and \(\mathcal{E}_{x_i, y_i} = 0\) otherwise, where \(x_i\) is the player’s move set and \(y_i\) is pholus’s move advice set in turn \(i\)⁴. Agreement is particularly useful for capturing the overlap when players reject the complete move advice set but follow individual advice from pholus.

3 Pholus Provides Helpful Advice↩︎

3.1 Quantitative Analysis↩︎

Non-advice factors parallel previous findings. Playing as France offers the most strategic advantage [8]. cicero playing as Germany or Italy is correlated with better game outcomes, while playing as Austria, England, or Turkey is correlated with worse game outcomes [4]. Additionally, cicero dominates: of twelve games, cicero won eight.

Advice helps. Playing a game without advice puts players at a disadvantage. The feature associated with no advice has a negative coefficient of approximately \(-0{.}05\) (Figure 2). The coefficients suggest a slight positive correlation between receiving move advice and point gains. Players who receive both move and message advice gain more points than those who receive only move advice. Interestingly, only having message advice negatively affect players’ game outcomes.

Novices can outperform experienced players with the help of pholus. Players with no prior experience in Diplomacy naturally face a disadvantage against seasoned players. This often results in novices being eliminated relatively early in the game. Even if they remain in the game, losing supply centers is almost inevitable. However, novice players receiving advice play better: in five games where novices received message and move advice, only one player was eliminated before the game concluded (typically \(3\)–\(4\) players in a game are eliminated). In the other four games, novices ended the game with more supply centers than they started with.

Novices are more likely to follow pholus’s advice. Experienced players tend to disregard advice. They accept only \(3.4\%\) of message advice and \(6.4\%\) of move advice from pholus. Although novice players are also hesitant to accept message advice, doing so \(6.3\%\) of the time, this rate is nearly double that of experienced players. Novice players follow move advice approximately one-third of the time, with an acceptance rate of \(32.6\%\). Both novice and experienced players tend to take more move advice than message advice.

Novices do not fully trust move advice from pholus. Across all games, pholus generates \(333\) instances of individual move advice for novices, organized into \(134\) sets. At the start of turns, average move agreement is \(80\%\) and average equivalence is \(46\%\), indicating strong alignment between novices’ initial idea for moves and pholus’s move advice. However, by the end of each turn, the average agreement drops by \(10\%\) and the average equivalence decreases by \(8\%\), indicating that novice players do not follow the move advice blindly.

3.2 Qualitative Analysis↩︎

While we can compute equivalence \(\mathcal{E}\) for moves, this is more difficult for messages. To better understand why players reject pholus’s advice more than they accept it (Table 1), we qualitatively investigate the differences between pholus’s suggested messages and the actual messages sent by players. To analyze message content, we use Abstract Meaning Representation [9] to extract Diplomacy-specific tokens. We parse player messages and the corresponding message advice from pholus to amr. We then measure the similarity of the two parses using smatch score [10]. Many pairs have high smatch, indicating that players often incorporate parts of pholus’s advice into their messages. For example, pholus suggests “bounce in Galicia again?” while the player wrote “Do you want to bounce in Galicia again?” Despite being written differently, these clearly have the same meaning, and indeed, smatch gives the pair a score of \(0.74\).

We also notice message-advice pairs with low smatch scores, where human players have different objectives in mind. For instance, in the fifth game, Italy captures Warsaw from Russia and anticipates losing it in the next turn due to Russia’s stronger nearby presence. When Russia inquires about the unexpected attack, pholus suggests using the fallacy of deflection to feign ignorance: “Turkey has been the only one to heed my concerns, despite my reservations,” and, “I thought you were lying”. However, the player disregards the advice of talking to Russia. Instead, the player seeks help from Turkey, who has an adjacent unit, to “support Warsaw’s hold”. The player then secures the support from Turkey and successfully keeps Warsaw from subsequent Russian attack. These pairs yield smatch scores of \(0\).

Our analysis indicates that smatch scores match with our intuitions about textual similarity. Given the many high smatch scores, we can conclude that many messages that are sent by players are minor variations on the provided advice. We provide more examples in Appendix 8.6. For additional qualitative insights, we survey players on the effectiveness of the advice. We summarize the survey results in Appendix 8.7.

4 Related Work↩︎

Appropriate Reliance on ai: The topic of human reliance on ai is central to current research in machine learning and explainable ai. Prior work measures reliance in ai-assisted decision making [11]–[14], and explores reducing over-reliance [15]–[17]. Some researchers have examined how explanations affect human reliance on ai [18], [19]. However, empirical evidence from multiple domains shows conflicting results: while some show that ai explanations improve human decision making, others find evidence of over-reliance on ai explanations even when they are incorrect [20]–[28]. For pholus, humans remain relatively conservative toward ai advice. Even novice Diplomacy players do not blindly follow the advice.

AI as Player Companion: AI agents have a long history of superhuman gameplay. In 1996, IBM‘s Deep Blue defeated the reigning world chess champion, Garry Kasparov, although it lost several other games in the same match [29]. More recently, DeepMind’s AlphaGo [30] consistently defeated top-rated Go players, a game with exponentially complex computational space, and later changed professional Go players’ play style. Multi-agent reinforcement learning systems like AlphaStar [31] and OpenAI Five [32] also show high performance in computer games through self-play.

However, these experiments focus only on game outcomes rather than how they can shape human gameplay. Some studies on nlp communicative agents aim to generate guidance in a grounded environment [33]. [34] develop an adaptive ai companion that adjusts its behavior based on the player’s experience. [35] assess human reliance on ai-based advice by examining the skill level of ai agents and the presentation of advice. While these studies show that ais outperform non-adaptive agents in guiding players, they do not consider player intention when generating guidance. In comparison, pholus takes players’ past messages and moves entered when generating personalized advice.

Augmented Learning: This is an educational approach that enhances and personalizes the learning experience. Traditionally, peer interaction simulates social interaction and helps learning [36]. Recent advancements in ai and nlp agents, suggest adaptive pedagogical interactions between humans and these agents to help learning in new environments [37]–[40]. [41] apply the theory of mind to generate guidance for players in Dungeons and Dragons. [42] develop a narrative-based tutoring system and show that it helps effective learning for children. In this study, we apply the concept of augmented learning to help novices understand the game of Diplomacy.

5 Conclusion↩︎

Human-ai collaboration depends on a range of factors. Using the board game Diplomacy, pholus provides real-time move and message advice tailored to intentions of both novice and experienced players. Surprisingly, even though only some advice is accepted, it can have a substantial impact on outcomes, particularly for novice players. This is because advice can positively inform choices even if the advice isn’t strictly followed. Our experiments enable further study of human-ai collaboration, including modeling explicit intentions and how to better use knowledge within these models. On a broader scope, future research should consider how ai can inform people without making choices for them and measure that impact.

6 Limitations↩︎

While we can effectively use pholus to generate both message and move advice for players, this advice can be too general or may not align with player intentions at times. For example, when a player expresses interest in an alliance with another player, pholus may give aggressive move advice deemed hostile toward that Power. We suspect that the advice may be optimized more for cicero’s intentions, which come from optimal moves in the supervised training data. Consequently, players who are willing to sacrifice individual optimality for mutual gains may find the advice less useful.

Furthermore, pholus cannot generate advice based on high-level player intentions. Specifically, pholus generates move advice based on optimal utility and message advice by inferring intentions from player-input moves. Potential improvements include 1) explaining meta-level intentions (e.g., ally with Germany and prioritize defeating Austria) from player input, and 2) generating targeted move and message advice based on meta-level intentions.

Finally, pholus is a resource-intensive advisor that runs on high-end gpus that require a large amount of on-chip memory (over 35GB). We use Nvidia’s A100 for running pholus. This limits accessibility for Diplomacy players and researchers to efficiently utilize pholus. The community would benefit from a distilled version of pholus by reducing computational limits and future adaptations.

7 Ethical Considerations↩︎

We recruit players individually via email and assign pseudonyms to ensure anonymity, even if players know each other outside the experiment. We adhere to human subject research regulations and the study was approved by our institution’s ethics review board (IRBNet ID: 1740681, University of Maryland). We report the experimental procedure in Appendix 8.3 and compensation details in8.4.

Acknowledgments↩︎

We thank Meta for open sourcing cicero. We thank the Diplomacy community for taking interest in our study. Specifically, our thank goes to Matthew Totonchy, Dr. Abhishek Singhal, Antonio Imperato, Sophia Wiste, and other members of the community who took the time to play against cicero. In addition, we thank Yanze Wang and Sadra Sabouri from University of Southern California for their helpful feedback. Denis Peskoff is supported by the National Science Foundation under Grant No. 2127309 to the Computing Research Association for the CIFellows 2021 Project. This research is supported by the U.S. Defense Advanced Research Projects Agency (DARPA) Other Transaction Award HR00112490374 from the Friction for Accountability in Conversational Transactions (FACT) program. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the view of our sponsors.

8 Appendix↩︎

8.1 Diplomacy↩︎

Diplomacy is a board game that has two core components: strategy and communication. Strategic reasoning plays a crucial role in determining the game’s outcome, as players’ moves directly impact the board’s status. Meanwhile, negotiation and deception significantly influence player strategies. Successful cooperation can remove a common adversary from the board, while a well-timed betrayal by a trusted ally can be catastrophic, greatly reducing the chances of winning. Excelling in Diplomacy requires not only a thorough understanding of the game’s mechanics but also strong communication skills. Consequently, Diplomacy is an ideal testbed for studying human-ai interaction and appropriate reliance in a grounded environment where outcomes are clearly observable.

Early efforts to develop agents for Diplomacy concentrated solely on creating rule-based agents that relied heavily on feature engineering [43]. These agents only submit moves and are not capable of communication. In 2002, a group of programmers released a communication protocol, Diplomacy Artificial Intelligence Development Environment  [44]. DAIDE defines a language syntax that enables agents to diplomatically negotiate and describe game actions. Following daide, researchers built communicative agents, including Albert [43], SillyNegoBot [45], DipBlue [46].

Starting with DipNet [7], neural networks were applied to the game, leading to the first agents that were competitive with people. Subsequent studies incorporated reinforcement learning to achieve super-human performance [47]–[50].

8.2 All Regression Coefficients↩︎

In Figure 2, we only show regression coefficients related to advice setting and player experience. Figure 3 contains coefficients for all regression features. The official Diplomacy rule states that supply center control changes only on even turns. However, we consider moving a unit to a center on an odd turn as gaining it, since the unit typically remains there in the next turn.

8.3 Experiment Procedures↩︎

The procedures involve playing Diplomacy with other participants as well as ai agents and providing feedback throughout the game on 1) the truth value of players’ messages 2) the predicted truth values of theirr opponents’ messages 3) the perceived friendliness of opponents 4) if players think they have been lied to in the previous phase. Players are asked to participate by placing their game moves in each turn before communicating. Some players in some turns received strategic moves and message advice from pholus. Games end after 3 hours or less. After each game, participants are asked to fill out a survey asking about their experience and the ai advice, as well as how ai agents performed during the game.

8.4 Participant Compensation↩︎

Players received a $70 gift card of their choice for each game. In addition, up to $29 was rewarded based on performance in the game.

8.5 Survey Questions↩︎

We use Google Forms to conduct player surveys. Table 2 shows the survey questions.

8.6 Pholus’s message advice and human messages with smatch scores↩︎

We provide additional examples of cases where human players reject pholus’s message advice, together with smatch. We show some advice (Figure 6 and 7) that experienced players mostly agree with, however, they partially edit to make it more aligned with their goals. We also show advice that is not aligned with the player’s goals, for example in Figure 8.

8.7 Survey Summarization↩︎

Many participants find pholus’s move advice helpful. Experienced players observe that the advice often aligns with their own moves and offers “some interesting ideas.” However, the advice can be suboptimal and short-sighted in complex scenarios, where it fails to consider interaction between allies. Players find message advice useful for simple, quick communication but inadequate for more complex or specific situations, especially when it does not align with their strategies or alliances. Players mention that the messages include common communication terms, and they “regret not using this feature more.” However, the advice is less helpful for specific planning and often does not align with player alliances and intentions.

References↩︎