Home > Uncategorized > Software engineering experiments: sell the idea, not the results

Software engineering experiments: sell the idea, not the results

A new paper investigates “… the feasibility of stealthily introducing vulnerabilities in OSS via hypocrite commits (i.e., seemingly beneficial commits that in fact introduce other critical issues).” Their chosen Open source project was the Linux kernel, and they submitted three patches to the kernel review process.

This interesting idea blew up in their faces, when the kernel developers deduced that they were being experimented on (they obviously don’t have a friend on the inside). The authors have come out dodging and weaving.

What can be learned by reading the paper?

Firstly, three ‘hypocrite commits’ is not enough submissions to do any meaningful statistical analysis. I suspect it’s a convenience sample, a common occurrence in software engineering research. The authors sell three as a proof-of-concept.

How many of the submitted patches passed the kernel review process?

The paper does not say. The first eight pages provide an introduction to the Open source development model, the threat model for introducing vulnerabilities, and the characteristics of vulnerabilities that have been introduced (presumably by accident). This is followed by 2.5 pages of background and setup of the experiment (labelled as a proof-of-concept).

The paper then switches (section VII) to discussing a different, but related, topic: the lifetime of (unintended) vulnerabilities in patches that had been accepted (which I think should have been the topic of the paper. This interesting discussion is 1.5 pages; also see The life and death of statically detected vulnerabilities: An empirical study, covered in figure 6.9 in my book.

The last two pages discuss mitigation, related work, and conclusion (“…a proof-of-concept to safely demonstrate the practicality of hypocrite commits, and measured and quantified the risks.”; three submissions is not hard to measure and quantify, but the results are not to be found in the paper).

Having the paper provide the results (i.e., all three commits spotted, and a very negative response by those being experimented on) would have increased the chances of negative reviewer comments.

Over the past few years I have started noticing this kind of structure in software engineering papers, i.e., extended discussion of an interesting idea, setup of experiment, and cursory or no discussion of results. Many researchers are willing to spend lots of time discussing their ideas, but are unwilling to invest much time in the practicalities of testing them. Some reviewers (who decide whether a paper is accepted to publication) don’t see anything wrong with this approach, e.g., they accept these kinds of papers.

Software engineering research remains a culture of interesting ideas, with evidence being an optional add-on.

  1. Nemo
    April 28, 2021 22:10 | #1

    And experimenting on university students should not be acceptable.

  2. April 29, 2021 12:24 | #2

    @Nemo
    There is nothing wrong in using students to beta test and experiment, but the results should not be published in any journal of repute. The reason is that the response characteristics of students are often different from professional developers.

  3. Nemo
    April 30, 2021 18:03 | #3

    Derek.
    Point taken (though the reason has always been “common knowledge” amongt those of us who hired people straight out of university.)

  4. John Cowan
    June 22, 2021 17:45 | #4

    Unfortunately, almost all published experimental social science is based on students from WEIRD (Western Educated Industrialized Rich Democratic) societies, who are very unlik the rest of humankind. See Henrich et al (2010).

  5. June 23, 2021 12:08 | #5

    @John Cowan
    True, but while WEIRD people may not be representative of the world as a whole, I think thay are representative of software developers. Henrich’s book The weirdest people in the world is a great read.

  1. No trackbacks yet.