Requirements for tools for comprehending highly specialized assembly language code and how to elicit these requirements

Abstract

Program comprehension tools used with assembly language—often for maintaining legacy software or reverse engineering malware threats—are dated and fail to provide rudimentary features found in tool support for higher-level languages. The need for people who can maintain these legacy systems is growing, as is the number of malicious cyberspace threats. To build new visualization and analysis tools within this domain, we need to understand the unique challenges faced by these developers. This paper presents the results of an exploratory case study to elicit requirements from two uniquely specialized groups of assembly language developers in an industrial setting: a large multi-national company developing mainframe software and a government defense facility analyzing malware and security flaws. In addition to surveys, observations and interviews, this study applies social psychology and nominal group techniques. We provide a ranking, and detailed description, for the requirements elicited in each group. We further include additional requirements obtained from observational studies. The ultimate conclusion we reach is that while similarities exist at a high level, upon deeper inspection, each group is quite unique with regard to their tooling needs.

Appendices

Appendix 1: Script used during the nominal group session

This script was used for the 2-h-long session with the Alpha group of six participants. While the same script was also used with the Beta group, times should be adjusted according to the size of the participant group.

Time	Action	Script
0-min introduction	SAY	Hi, I’m RESEARCHER NAME from UNIVERSITY NAME. For my PhD in Computer Science, I am exploring how visualization and tool support for assembly language might be useful. My work is being funded by COMPANY NAMES
		Since you are experts in the area, we really value your experience and expertise in defining the issues
		This is SECOND RESEARCHER NAME and I’ll let him introduce himself
		University
		Degree
		Research interest
		To get started, I’d like to collect the ethics forms that you were given yesterday
	DO	Collect the ethics forms
	SAY	This session should take no longer than 2 h, including a 20 min break. The aim is to discuss and critically rank all of the items from the exercise yesterday. If you come up with new ideas during the session, please add them to your list. Feel free to be creative
		Does everyone have the blue pages?
		First of all, I’d like to go around the table and have everyone introduce themselves and tell us about your job. We also know from the survey that your teams are expertise-centered, so it would be great to hear about that, as well as your interests
10-min listing of ideas	SAY	Now to begin the group exercise, we will go around the table and each person will share one item from their list at a time. At this time, please avoid discussion or talking out of turn
		After all of the items are listed, we will have a discussion to clarify the items. If you have any new ideas then feel free to add them to your sheet. If you want to skip a turn, that is also fine
	DO	Record word for word what each person says on the power point slide
30-min discussion of ideas	SAY	We will now have a 30 min discussion on all the ideas generated
		Now is the time to ask for clarification or elaboration on an idea, or dispute or defend an item
		You are also welcome to suggest new items during this time, but no items can be eliminated
		We’ll go through them item by item
	DO	Announce each item on the list and ask what it means, or how people feel about it. Record any new ideas on the power point slide
60-min ranking to select the “top ten” ideas	SAY	Now if everyone could take out their yellow sheet for preliminary ranking
		You can see there are 10 spaces to be filled in. You can select 10 items that are the most important for you from all of the options. Then assign them a rank which is a numbering between 1 and 10, where 10 is the most important
		Once you are finished, please turn it face down on the table and then you are free to take a break for about 20 min
70-min break	DO	Go around the table and transcribe and sum up the points from the ranking sheets onto the power point slides. Then reorder them on the slide based on the greatest number of points
		Collect everyone from after their break
90-min discussion of vote	SAY	We have reordered the items according to rank and you can see the score for them. We have also highlighted the top ten
		We will now have a free-for-all discussion about the nature and content of the top ten
		We would also like to hear how you feel about items that should have been included or excluded from this list
110-min re-ranking and rating revised “top ten” items	SAY	Now if everyone could take out their green sheet for final ranking. Here you will again list the top ten items that you think are the most important
		This may be the same ten, or feel free to modify which items are in your top ten
		The ranking here is different in that 100 points will be given to the most important item. Every other item can have a value between 0 and 100. Two items can have the same ranking
		Once you are finished, please hand in your sheets to me face down, and then we’re all done!
	DO	Collect the green sheets from everyone and tally up the final scores based on the 0–100 ranking
		END CASE STUDY AT (START + 120 MIN)

Appendix 2: Issues observed at the Alpha group during activity-based protocol elicitation

Requirement category	Issue	Description
First session
Browsing and navigation	XREF works on only 8 character long names	When there are more, search must be used, which only finds them one at a time in the code
	Bookmarking lines of code	Have to create names “a,” “b.” If the name already exists, it is just overwritten
	Lack of navigation	Need to scroll through many screens of code to look for the right spot
Build
Control flow	Hard to find main task
	Tools would need to support multi-threading
Data
Debugging	Timing issues were tricky	Timing dumps not useful because they are too complicated
	Couldn’t work out what was causing the cancel	Need some way to trap the event
	XREF plus debugger to find the correct place to debug	Step-through debugging might be helpful
De-obfuscation	Redundant code makes the code confusing to read	Statements such as branching to the next address. Unnecessary since that code is next to be executed
Documentation	Look up vendor error code in CA documentation	Not indexable online so need to download CA docs to search them. CA error code is then used to look up IBM Manual error code. Codes are OS version dependent
	User prints off whole modules	The printoff is portable and more comfortable to look at (easier on the eyes). There are also sticky notes and writing on the pages. These written notes include variable names, addresses and error codes
	The dump was scrolling off the page	There were so many errors, it did not fit. Need a way to condense it
Integration
References
Source control	Object module replacement	Overwrites whole module, have to be careful not to overwrite a change. Have to check prerequisite chain, and which fixes supersede others
Source editing
Second session
Browsing and navigation	*temp is used as a TODO	Shows up only when you dig into the module you’re interested in. Used pdsman to scan and find. Scan doesn’t show the active module however
	Switching terminal screens constantly	Need to scroll through many screens of code to look for the right spot. Kept many terminal screens open. Was hard to keep track of which showed the right code
Build	Register usage	Waits for compile error to say that the register is in use
	? at the start of lines	To ensure you get errors, but do no want to deal with the actual errors (stub error)
	Scanning software for changes he knows he has to make	Otherwise waits for compile errors. Compile errors would be better if they occurred during editing. Context aware correction suggestions (i.e., does not exist, did you mean…?). Calls out to code that does not exist anymore
Control flow
Data
Debugging	No breakpoints in XDC	Puts code in to make it fail
De-obfuscation
Documentation	IBM Principles of Hardware Manual	Useful to double check some things
Integration
References	Code module—fan in, fan out	Wanted to know what module was being called dependent on the code and what code it depended on
Source control
Source editing	Tedious refactoring of modules	Splitting larger modules into smaller ones to use as templates. Templates are not useful, not maintained that much, but useful for people starting from scratch. Instead he uses something else he’s working on, copies it and butchers it (side by side editing)
	Forgot to save the file	No alert was given
	Code shortcuts	Stuff he does more than once