Algorithms for Automatic Ranking of Participants and Tasks in an Anonymized Contest

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10167)


We introduce a new set of problems based on the Chain Editing problem. In our version of Chain Editing, we are given a set of anonymous participants and a set of undisclosed tasks that every participant attempts. For each participant-task pair, we know whether the participant has succeeded at the task or not. We assume that participants vary in their ability to solve tasks, and that tasks vary in their difficulty to be solved. In an ideal world, stronger participants should succeed at a superset of tasks that weaker participants succeed at. Similarly, easier tasks should be completed successfully by a superset of participants who succeed at harder tasks. In reality, it can happen that a stronger participant fails at a task that a weaker participants succeeds at. Our goal is to find a perfect nesting of the participant-task relations by flipping a minimum number of participant-task relations, implying such a “nearest perfect ordering” to be the one that is closest to the truth of participant strengths and task difficulties. Many variants of the problem are known to be NP-hard.

We propose six natural k-near versions of the Chain Editing problem and classify their complexity. The input to a k-near Chain Editing problem includes an initial ordering of the participants (or tasks) that we are required to respect by moving each participant (or task) at most k positions from the initial ordering. We obtain surprising results on the complexity of the six k-near problems: Five of the problems are polynomial-time solvable using dynamic programming, but one of them is NP-hard.


Chain Editing Chain Addition Truth discovery Massively open online classes Student evaluation 



This work was supported in part by the US National Science Foundation under award numbers CCF-1527032, CCF-1655442, and IIS-1553547.


  1. 1.
    Agrawal, A., Klein, P., Ravi, R.: Cutting down on fill using nested dissection: provably good elimination orderings. In: George, A., Gilbert, J.R., Liu, J.W.H. (eds.) Graph Theory and Sparse Matrix Computation, pp. 31–55. Springer, Heidelberg (1993)CrossRefGoogle Scholar
  2. 2.
    Andersen, R., Borgs, C., Chayes, J., Feige, U., Flaxman, A., Kalai, A., Mirrokni, V., Tennenholtz, M.: Trust-based recommendation systems: an axiomatic approach. In: WWW, pp. 199–208. ACM (2008)Google Scholar
  3. 3.
    Aydin, B., Yilmaz, Y., Li, Y., Li, Q., Gao, J., Demirbas, M.: Crowdsourcing for multiple-choice question answering. In: IAAI, pp. 2946–2953 (2014)Google Scholar
  4. 4.
    Balas, E., Simonetti, N.: Linear time dynamic-programming algorithms for new classes of restricted TSPs: a computational study. INFORMS J. Comput. 13(1), 56–75 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Bertsekas, D.P.: Non-linear Programming. Athena Scientific, Belmont (1999)Google Scholar
  6. 6.
    Bliznets, I., Cygan, M., Komosa, P., Mach, L., Pilipczuk, M.: Lower bounds for the parameterized complexity of minimum fill-in and other completion problems. In: SODA, pp. 1132–1151 (2016)Google Scholar
  7. 7.
    Cao, Y., Marx, D.: Chordal editing is fixed-parameter tractable. Algorithmica 75(1), 118–137 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Cao, Y., Sandeep, R.B.: Minimum fill-in: inapproximability and almost tight lower bounds. CoRR abs/1606.08141 (2016).
  9. 9.
    Dong, X.L., Berti-Equille, L., Srivastava, D.: Integrating conflicting data: the role of source dependence. PVLDB 2(1), 550–561 (2009)Google Scholar
  10. 10.
    Drange, P.G., Dregi, M.S., Lokshtanov, D., Sullivan, B.D.: On the threshold of intractability. In: ESA, pp. 411–423 (2015)Google Scholar
  11. 11.
    Feder, T., Mannila, H., Terzi, E.: Approximating the minimum chain completion problem. Inf. Process. Lett. 109(17), 980–985 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Fomin, F.V., Villanger, Y.: Subexponential parameterized algorithm for minimum fill-in. In: SODA, pp. 1737–1746 (2012)Google Scholar
  13. 13.
    Galland, A., Abiteboul, S., Marian, A., Senellart, P.: Corroborating information from disagreeing views. In: WSDM, pp. 131–140. ACM (2010)Google Scholar
  14. 14.
    Gatterbauer, W., Suciu, D.: Data conflict resolution using trust mappings. In: SIGMOD, pp. 219–230 (2010)Google Scholar
  15. 15.
    Gupta, M., Han, J.: Heterogeneous network-based trust analysis: a survey. ACM SIGKDD Explor. Newsl. 13(1), 54–71 (2011)CrossRefGoogle Scholar
  16. 16.
    Jiao, Y., Ravi, R., Gatterbauer, W.: Algorithms for automatic ranking of participants and tasks in an anonymized contest. CoRR abs/1612.04794 (2016).
  17. 17.
    Kaplan, H., Shamir, R., Tarjan, R.E.: Tractability of parameterized completion problems on chordal, strongly chordal, and proper interval graphs. SIAM J. Comput. 28(5), 1906–1922 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Kleinberg, J.M.: Authoritative sources in a hyperlinked environment. JACM 46(5), 604–632 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Li, Q., Li, Y., Gao, J., Zhao, B., Fan, W., Han, J.: Resolving conflicts in heterogeneous data by truth discovery and source reliability estimation. In: SIGMOD, pp. 1187–1198 (2014)Google Scholar
  20. 20.
    Li, Y., Gao, J., Meng, C., Li, Q., Su, L., Zhao, B., Fan, W., Han, J.: A survey on truth discovery. ACM SIGKDD Explor. Newsl. 17(2), 1–16 (2015)CrossRefGoogle Scholar
  21. 21.
    Natanzon, A., Shamir, R., Sharan, R.: A polynomial approximation algorithm for the minimum fill-in problem. SIAM J. Comput. 30(4), 1067–1079 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Pasternack, J., Roth, D.: Knowing what to believe (when you already know something). In: COLING, pp. 877–885 (2010)Google Scholar
  23. 23.
    Pasternack, J., Roth, D.: Latent credibility analysis. In: WWW, pp. 1009–1021 (2013)Google Scholar
  24. 24.
    Pasternack, J., Roth, D., Vydiswaran, V.V.: Information trustworthiness. AAAI Tutorial (2013)Google Scholar
  25. 25.
    Wu, Y.L., Austrin, P., Pitassi, T., Liu, D.: Inapproximability of treewidth, one-shot pebbling, and related layout problems. J. Artif. Int. Res. 49(1), 569–600 (2014)MathSciNetzbMATHGoogle Scholar
  26. 26.
    Yannakakis, M.: Computing the minimum fill-in is NP-complete. SIAM J. Algebr. Discret. Methods 2(1), 77–79 (1981)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Yin, X., Han, J., Yu, P.S.: Truth discovery with multiple conflicting information providers on the web. TKDE 20(6), 796–808 (2008)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Tepper School of BusinessCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations