Skip to main content

Reflections on the Empirical Software Engineering journal

Victor Basili (1996–2014):

In the 1994–1995 time period, I was contacted by Kluwer Academic Publishers because they had been talking with Warren Harrison and were interested in creating a journal on software measurement and wanted to know if I was interested in co-editing it. I had recently stepped down from the Editor in Chief (EiC) position of IEEE Transactions on Software Engineering a couple of years earlier but was still an associate editor of the Elsevier Journal of Systems and Software. So, although I was not interested in taking on more editorial work, I suggested that I would be interested if it were a “Journal of Empirical Software Engineering” as I thought the software measurement was too narrow a topic and felt that making the domain empirical and experimental engineering would open the journal to what was a new and important area of research and would give that area a home. My argument had been that although there was enough work currently underway to create a journal, the area would surely grow over time because it was a natural evolution in any scientific area. And although it was the beginning, it could be a repository of papers that would allow people to easily find what work was currently going on in the area.

There had already been several important papers including the work of Les Belady and Manny Lehman, Barry Boehm, William Hetzel, and Gerald Weinberg, and others who were building models and evaluating the effectiveness of technologies. They ranged from work done with data from various companies to small experimental studies at universities. But they were isolated events. At the time, I had been working in both types of studies: working with NASA Goddard on building models of various aspects of software development and experimenting with technologies in small, controlled studies with students. But I believed we needed to establish an environment for people to build upon each other’s results and identify the causes of the differences in results.

It is also true that the whole idea of empirical studies in software engineering was not embraced by everyone. For example, I had published a paper, with Robert Reiter, evaluating the effects of technology in small classroom studies in IEEE Transactions on Software Engineering, that won the best paper award for that journal in 1981. And to make it clear how little empirical work was understood, a later reviewer of the paper wrote that the study wasn’t necessary because he would have predicted results, so there was no need for the study, i.e., there was no need for empirical evidence.

A second issue was that the papers that were out there were published in a variety of venues and it was not easy to find them. Empirical study requires multiple studies and opportunities for comparison of the results. It needs a community.

So that was the original motivation behind the journal. Now came the business of making it work. We needed a collection of people who were willing to be on the editorial board of the new journal, i.e., who believed in the idea and were willing to put in the effort of reviewing and advising authors.

Collecting an initial Editorial Board for the Journal of Empirical Software Engineering was not such a difficult job. About a third of the new editors were collaborators or students of mine. We had an editorial board of 25, representing 4 continents, 12 academic institutions, and 11 industrial organizations. The goal was to start small and publish the journal quarterly. In the first year, Volume 1 (1996), we only published three issues of the journal that contained nine peer reviewed papers, one viewpoint and three editorials, for a total of 305 pages.

We wanted to make the journal a major resource for people doing empirical software engineering work. So, we included editorials by the editors but also by other contributors, viewpoints to generate discussion, selected papers from conferences, dissertation abstracts, and special sections on a particular topic that would only take part of the available pages. We did not devote an entire issue to a special topic.

Another goal was to have fast turn around from submission to appearance in print. We had a goal of six to eight months, which we did not always make but always tried.

By the sixth volume (2001) we had grown to 367 pages, and Warren decided to move on to other activities, so, we invited Lionel Briand to take on co-EiC duties and in volume 7 we had three EiCs. From volume 8 on, we had two editors-in chief, Lionel and me. From the beginning Lionel had great ideas and enormous enthusiasm.

And so, the journal continued to grow slowly. In Volume 7 (2002) we had 284 pages; in Volume 8 (2003) there were 396 pages, Volume 9 (2004) there were 374 pages. Volume 9 was that last volume published by Kluwer. Springer bought the journal from Kluwer.

Volume 10 (2005) had 536 pages. And by then we had 41 editors. As issues were getting thicker, we decided with Springer to go to six issues a year. This allowed us to shrink the time to publication and minimize the size of an issue, but this did not last for long.

By the time we got to Volume 11 (2006), the journal ran 611 pages in 6 issues per year. By Volume 14 (2009) the journal ran 778 pages with over 50 editors.

2014 was my last issue as co-EIC. I had retired from the University of Maryland and had not been pulling my share of the load for a couple of years. As the number of papers published a year was growing enormously, we agreed to find a new second editor-in-chief to take my place and offered the position to Thomas Zimmermann.

I leave it to Lionel, Thomas, and Robert to add their goals and experiences to the journal history. But I have to say my goal has been accomplished. Empirical Software Engineering has expanded and matured tremendously over the last two decades, as a field and as a journal. I was lucky to be in the early stages of the field and I believe that the journal helped make that maturing happen. The journal and the other activities, such as ISERN, where researchers can meet, interact and exchange ideas, have had a strong effect on the field. and the creation of a community. Now, no one questions the need to do empirical work and to collaborate with organizations that build systems. Over the last couple of decades, empirical work has gained respect and has had the ability to demonstrate its importance to software engineering.

figurea

Lionel Briand (2002–2017):

In 2002 as I got increasingly involved, under the guidance of Vic Basili, in the management of the journal, I asked myself what objectives I wanted to achieve. Empirical software engineering was by then a largely accepted field of study and empirical work was widely published in other venues. However, at that point, the journal was not yet considered a top-tier publication venue but a rather specialized outlet. Given that empirical methods were cross-cutting all aspects of software engineering research and by now a widely accepted means of conducting research, that potentially put into question both the need for such a journal and its role in the community.

In the meantime, what I had noticed was an increasingly widening gap between research and practice. Though empiricism was at the heart of what the journal stood for, I wanted its role to expand and promote applied, industry-focused research, as publishing such work was clearly still a severe challenge in prominent journals and conferences. I considered this type of work to be fundamental in an engineering discipline, as I had been strongly influenced by Vic’s SEL lab and its close collaboration with the NASA Goddard flight center. I wanted us to play a central role in strengthening that research paradigm and the acceptance of such work in academic research. Empirical methods, through case study research, qualitative studies, and mining software repositories, were the fundamental instrument by which to achieve academia-industry collaborations and produce impactful research in software engineering. In other words, we wanted to re-define the scope of the journal to encompass any applied, impactful software engineering research with a strong empirical component.

Another ambition was to transform the journal into a first-tier and general publication venue addressing all subfields of software engineering, from requirements engineering to testing and evolution. This had never been the case for any non-IEEE and ACM journal in the past. The challenge was therefore steep. I asked myself what it would take to achieve that and concluded that there was no quick solution, that changing the perception about our journal required a long-term, persistent strategy. Everything in our profession is about perception and that is indeed always the first aspect to focus on. My strategy was to become the best managed journal in the field, handling the review process with rigor and interacting with authors at the highest standards of professionalism. The hope was that the journal, through people’s experience both as authors and readers, would be increasingly perceived as focused on quality and impact. We drafted guidelines for associate editors to follow and carefully monitored their application. Further, we expanded the editorial board to fully cover the various expertise areas of the field and to ensure that associate editors had a reasonable workload and could dedicate the required time to each paper. Associate editors were seen as an essential part of the review and decision process, relying on reviews for inputs, but with a role going far beyond the mere selection and coordination of reviewers.

By 2016, I felt my job was done and that new energy and fresh perspectives were required to bring the journal to a new level and solidify its stature. Tom and I invited Robert Feldt to become EiC and replace me. By the time I had left the journal as EiC in 2017, Empirical Software Engineering had become recognized as one of the very best journals in software engineering. Reflecting this, it was selected as one of the three journals participating into the Journal First initiative at ICSE, along with IEEE and ACM Transactions. At that point, the journal was handling a very large volume of submissions and published 91 articles in 2017, a high number by any standards. Many of these published articles focused and reported on industrially-relevant research.

figureb

Author information

Affiliations

Authors

Corresponding authors

Correspondence to Victor Basili or Lionel Briand.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Basili, V., Briand, L. Reflections on the Empirical Software Engineering journal. Empir Software Eng 27, 6 (2022). https://doi.org/10.1007/s10664-021-10055-9

Download citation