1 Introduction

1.1 Is High-Frequency Trading Fintech?

High-frequency trading (HFT), which has been actively practiced in the US since the early 2000s, began spreading more widely in Japan around 2010. Today, after some ten years, opinion is still divided regarding the impact of HFT on Japanese financial markets, and many aspects of its effects have yet to be fully evaluated. HFT defies straightforward judgment, due to a marked lack of clarity regarding matters such as the actual status of HFT activity, its effect on financial markets and the possible existence of unfair trading.

Does AI-driven algorithmic trading, together with HFT as one of its subclasses, really constitute fintech? If fintech is understood in the broadest sense as a fusion of finance and technology, then algorithmic trading and HFT are undoubtedly fintech fields. However, defining fintech more narrowly as innovations in financial services that enhance the convenience of end users creates greater ambiguity about whether or not HFT is actually fintech, because it is at best only experienced by end users of financial services indirectly.

1.2 Recognizing the Social Significance of HFT

In the author’s understanding of public discourses, many people seem to have a negative impression of HFT, such as “a way for only a few market participants to make money,” or believe that “advances in HFT technologies have not led to win–win situations that benefit more participants, but rather to zero-sum games,” or that “HFT manipulates the market and harms the interests of other investors, particularly individual investors.”

Professor Joseph E. Stiglitz of Columbia University concluded that, while HFT firms profit from making trades faster than other firms do, such activity leads to excessive investment and wasted costs in social terms. Moreover, the involvement of HFT is suspected each time a “flash crash” occurs, where prices in equities, bonds, or Forex markets fluctuate significantly over a short space of time. Consequently, HFT is often criticized for having a destabilizing influence on financial markets (Financial Services Agency, 2016).

Nonetheless, considerable empirical analysis in Japan and throughout the world indicates that HFT enhances market performance, including increasing the liquidity and improving the efficiency of markets (Brogaard et al., 2014a; Benos & Sagade, 2016; Zhang & Riordan, 2011). There is a far greater volume of research demonstrating HFT’s positive impact than empirical analyses showing that HFT destabilizes financial markets. It may, therefore, be reasonable to suggest that HFT has broad social significance, benefitting society as a whole by enhancing market functioning. The importance of this impact should not be underestimated.

1.3 The Movement Toward Greater Regulation of HFT Around the World

Even if HFT clearly enhances market performance under normal conditions, can the same be said in the case of a crisis or an emergency? It is arguable that if markets become turbulent for some reason, HFT may actually amplify this instability. To date, however, there has not been sufficient evidence of this effect.

In addition, there is good reason to believe that HFT firms may, deliberately or unintentionally, as a result of their nature of operation, be engaged in unfair trading practices, such as market manipulation (Niwa, 2016). It is also possible that trading by HFT firms may deprive other investors, particularly individual investors, of opportunities to gain profits. These issues, which are focusing on market efficiency, but are more concerned about the distribution of profits, still await thorough evaluation.

Throughout the world, countries have proceeded to strengthen regulation and implement systems to respond to the risks potentially associated with HFT. In Japan, for example, a registration system for HFT firms was instituted in April 2018.Footnote 1

The regulation of HFT requires sophisticated technology, and represents a new frontier for regulators. In fact, the level of technology used by regulators has been cited as one reason why, in Japan, very few cases of unfair trading by HFT firms have been exposed to date. In order to improve the level of technology available to regulators, more collaboration with the private sector will be needed.

It is to be hoped that this increase in regulation will not overly constrain those HFT activities that contribute to the public good, including its improvement of market functioning, but rather mitigate risks that may result in the potential problems described above.

1.4 HFT May Play a Role in Shaping Business Models in Japan's Securities Industry

Included in this chapter’s review of the latest trends in the activities of Japan’s securities companies and HFT firms is an examination of the mechanism, already common in the US, whereby an online securities broker may pass on share trading orders submitted by individual investors to an HFT firm in return for compensation in the form of a rebate from the HFT firm. Japanese online securities brokers have begun adopting this practice, leading to an increasingly strong relationship of mutual dependence between HFT firms and securities companies.

Japan’s online securities brokers face an extremely fragile earnings base due to a persistent low interest rate environment. In the future, it is conceivable that these brokers may, therefore, grow even more dependent than their US counterparts may on rebate income from HFT firms.

Continued, strict monitoring will be necessary to ensure that these practices are not conducted in ways that significantly damage the interests of individual investors.

2 Algorithmic Trading and HFT

2.1 Algorithmic Trading

HFT is one form of algorithmic trading. Algorithmic trading can be defined as the repeated trading of securities where the timing and volume of orders placed is determined automatically by a computer system, according to a predesignated procedure.Footnote 2 The main objective of algorithmic trading is to achieve stable profits. To this end, it seeks to pursue maximum returns while controlling risk and reducing costs.

Algorithmic trading itself has been used for quite some time. In many cases, it is not particularly sophisticated, consisting of nothing more than the automation of conventional trading procedures. Indeed, a significant proportion of algorithmic trading is not actually high-frequency, high-speed trading. Recently, however, there has been an increase in sophisticated algorithmic trading utilizing AI technologies based on machine learning.

Institutional investors, proprietary firms such as investment companies that invest using only proprietary funds for direct gains rather than for commissions, proprietary trading and brokerage divisions of securities companies, and even individual investors all engage in algorithmic trading. For all of these players, with the exception of the brokerage divisions of securities companies, the main objective of algorithmic trading is achieving maximum profits. For the brokerage divisions of securities companies, the main objective is fulfilling their duty of best execution; that is, their obligation to ensure that customer orders are executed under the best possible conditions.

2.2 Types of Algorithmic Trading

Algorithmic trading can be classified into six types according to its objective and procedure: (1) execution algorithms, (2) benchmark execution algorithms, (3) market-making algorithms, (4) arbitrage algorithms, (5) directional algorithms, and (6) market manipulation algorithms (NTT DATA Financial Solutions Corporation, 2018).

  1. (1)

    Trading using execution algorithms

    Algorithms for executing trades automate the splitting and timing of buy or sell orders placed by investors, choose optimal markets, and make other adjustments, in order to achieve objectives such as cost reduction. Some of these algorithms are designed to conceal the execution of trades from other investors, thus mitigating market impact cost, i.e., the price change that occurs from the action of buying or selling a security. Others incorporate mechanisms to ensure compliance with market rules.

    Splitting large orders into smaller ones, and placing these smaller orders gradually over time, is an effective way of reducing market impact cost. However, the longer it takes to complete the execution of an order, the greater the risk of market price movements (timing cost). Therefore, one important role of execution algorithms is to determine and implement the optimal timing that will minimize the sum of these two costs.

  2. (2)

    Trading using benchmark execution algorithms

    Benchmark execution algorithms, aimed at ensuring that the results of order execution approximate a defined benchmark, are used when executing large orders. For example, when splitting a large order into several smaller ones in order to limit market impact cost, a benchmark execution algorithm may be designed and applied to ensure that the average price of each small order approximates a benchmark such as the market closing price.

  3. (3)

    Trading using market-making algorithms

    Just as regular market makers do, market-making algorithmic traders place both buy and sell limit orders. By placing such orders simultaneously, at prices more favorable than the current market price (mid-price), and then awaiting other market participants to trade with, market-making algorithmic traders aim to profit from the difference between the market price and the bid or ask price. If buy and sell orders of the same size are executed, the trader will then profit from the combined bid–ask spread.

    Such trading by market makers provides market liquidity, thus contributing to the stability of markets. Investors utilizing market-making algorithms must constantly adjust spreads and order sizes to respond to the movements of markets and order books, repeatedly place new orders, adjust, and cancel orders in accordance with these changes.

  4. (4)

    Trading using arbitrage algorithms

    When the prices of identical securities, or other equally valued products or instruments, differ at the same point in time, arbitrage algorithms seek to generate profits by simultaneously selling at the higher price and buying at the lower price, and then closing these positions after the prices converge. In this way, traders can make profits while limiting the price change risk (market risk). To the extent that the application of arbitrage mitigates or eliminates distortions in markets, it can be said to contribute to enhancing market efficiency.

    Four processes must be completed before arbitrage trading can generate profits: the discovery of arbitrage opportunities; the opening of arbitrage positions; the total or partial resolution of price distortions; and the closing of the arbitrage positions. Because the effect of arbitrage is to eliminate price distortions, the investor who first takes advantage of an arbitrage opportunity can make the greatest profit. Therefore, speed is vital in the first and second arbitrage processes: the discovery of arbitrage opportunities, and the opening of arbitrage positions, respectively.

  5. (5)

    Trading using directional algorithms

    Directional algorithms are used to predict changes in market prices using market data such as prices and trading volumes as well as news and other event data. They are also used to generate profits from trading based on these predictions. The strategy behind their use is to profit from unidirectional changes in market prices. This style of trading is generally high-risk and high-return.

  6. (6)

    Trading using market manipulation algorithms

    Market manipulation algorithms are applied to move market prices in a favorable direction by issuing orders designed to mislead other market participants with respect to information, such as the provision of liquidity or the intention to buy or sell. Using these algorithms can enable the user to achieve considerable profits. In some cases, these algorithms can operate to reduce trading costs by attracting significant liquidity to the market. They can also delay or prevent the execution of orders by other market participants by causing the repeated cancellation of large orders.

2.3 Using Machine Learning in Algorithm Construction

Two types of methods are used to construct the algorithms used in the trading strategies described above; a theoretical approach and an empirical approach (NTT DATA Financial Solutions Corporation, 2018). Using the theoretical approach, the designer establishes certain assumptions regarding price movements and the mechanisms that determine market conditions, and constructs a model based on this. In contrast, using the empirical approach, a computer is programmed to discover patterns in historical data using AI technologies, such as machine learning, and then search for a model that matches these patterns.

The theoretical approach facilitates the validation of the assumptions made and the correction of any problems, as the designer understands the mechanism of the algorithm. At the same time, however, the strength of the theoretical approach is dependent on the designer’s individual experience and is constrained by the fact that there is a limit to the number of theoretical causal relationships that any designer can recognize and understand. Consequently, it can be anticipated that using the empirical approach to construct models based on more extensive and diverse case data will lead to better trading performance. Therefore, a combination of theoretical and empirical approaches is often used in algorithm construction.

2.4 The Struggle Between AI Technologies

Among the forms of algorithmic trading described above, competition often arises between the AI technologies used by the brokerage divisions of securities companies in their execution algorithms, and market-making algorithms used by HFT firms. The execution algorithms used by securities companies automatically determine a series of processes for the execution of large orders received from customers, such as order splitting, order timing adjustment, and the selection of optimal markets. In this way, execution algorithms try to prevent these large orders from being detected by other investors, and to execute them without giving rise to market price movements. In contrast, HFT firms using market-making algorithms aim to profit from rapidly placing, altering, and canceling both buy and sell orders. Rather than preventing the detection of large orders, as execution algorithms do, these market-making algorithms operate to quickly detect the existence of large orders in the market and then profit by anticipating their execution. This leads to an intense struggle between the AI technologies designed to conceal the existence of large orders, and the AI technologies designed to uncover them.

2.5 What is HFT?

HFT refers to a type of algorithmic trading where securities are bought and sold at high speed and high frequency. The Committee on Economic and Monetary Affairs of the European Parliament (2012b) defines high-frequency trading as “algorithmic trading in financial instruments at speeds where the physical latency of the mechanism for transmitting, canceling, or modifying orders becomes the determining factor in the time taken to communicate the instruction to a trading venue or to execute a transaction” (art. 4, para. 2(30a)). The Committee further characterizes a high-frequency trading strategy as a trading strategy that involves high-frequency trading and satisfies two of the following five conditions:

  1. (i)

    The utilization of co-location services (services that allow trading participants to place servers and other devices that execute trades physically close to the trading system operated by the securities exchange), direct market access or proximity hosting

  2. (ii)

    The daily trading value is at least 50% of the portfolio

  3. (iii)

    The order cancellation rate is higher than 20%

  4. (iv)

    The majority of positions taken are unwound within the same day

  5. (v)

    There are discounts or rebates on more than 50% of transactions or orders (art. 4, para. 2(30b)).

Trading algorithms used in HFT

Of the six forms of trading algorithms examined above, three in particular tend to be used in HFT: market-making algorithms, arbitrage algorithms, and directional algorithms. The most common of these are market-making algorithms. High-frequency, high-speed trading is effective for market-making, because of the need to constantly place, alter, and cancel orders according to changes in market prices and liquidity.

For arbitrage, the greatest profits can be generated by algorithms that are able to discover price distortions—arbitrage opportunities—and execute arbitrage trades the fastest. In this context too, the use of HFT is effective. This type of algorithmic trading, HFT, was described in Flash Boys: A Wall Street Revolt by Michael Lewis, published in 2014. In the US, New York is the hub for trading individual stocks, while trading of equity index futures is centered in Chicago. A direct fiber-optic cable was laid between these two cities with the aim of encouraging arbitrage trades between their two markets.

Similarly, in the case of directional algorithms, the use of HFT is effective when the aim is to attain trading profits over a short period of time.

2.6 Background to the Growth of HFT for Arbitrage in the US

Regulatory reform in the US provided the opportunity for more active use of HFT in arbitrage. The US Securities and Exchange Commission (SEC), uneasy about the monopoly exercised over equities trading by the New York Stock Exchange (NYSE) and Nasdaq, promoted regulatory reform aimed at stimulating competition between securities exchanges. As a result, from the 1990s onward, markets became increasingly fragmented, with orders executed on a greater number of exchanges or alternative trading systems (ATS), or by market makers other than exchanges.

The more places—markets—where a stock is traded, the greater the number of possible discrepancies between indicative prices and, therefore, the greater the opportunity for arbitrage. Investors progressively introduced high-speed trading systems capable of rapidly responding to changes in order book information. At the same time, markets, i.e., securities exchanges, themselves also increased the response speed of their order execution systems in order to meet the needs of these investors.

In Japan, however, with the Tokyo Stock Exchange accounting for around 90% of the total value of trades, the use of HFT for arbitrage is relatively minimal. Rather, the use of HFT in Japan centers on market-making algorithms.

2.7 Will HFT Approach the Speed of Light?

A relatively small number of emerging companies manufacture network switches that enable the processing of transactions at the equivalent of the speed of light. In 2016, The Wall Street Journal reported that network switches manufactured by Metamako, based in Sydney, Australia, and xCelor, based in Chicago, required just four nanoseconds, i.e., four billionths of a second to relay information such as data sent from a securities exchange to an electronic trader (Sprothen, 2016). Consequently, for some HFT processes, trading really is approaching the speed of light. Does this mean that the competition for greater speeds in HFT is coming to an end?

As the speed of trading almost literally approaches the speed of light, the amount of investment required to increase this speed so that it is even fractionally higher than that of competitors is growing exponentially. With the marginal cost of greater speeds becoming higher, HFT firms can be expected to stop making additional investments in speed when the marginal cost of such investments matches the marginal expected return. As I will discuss later, the proportion of HFT within all equity trading in the US has actually been decreasing since its peak around 2009. Some have cited this as an indication that the investment in speed has already reached just such a critical point at which the cost of such investment is no longer worthwhile. Nevertheless, in the US, firms still compete to achieve speeds even fractionally faster than that of their competitors moving ever closer to the speed of light. Clearly, the critical point has not yet been reached.

3 Reviewing the Historical Development of HFT in Japan and Around the World

3.1 HFT First Flourished in the US

It was in the US that HFT first became popular. By the mid-2000s, many HFT firms were already participating in US markets. The percentage of HFT in all stock trades increased rapidly through the second half of that decade, and had reached 61% by 2009, according to an estimate by Valerie Bogard of the Tabb Group, a US research firm (Bogard, 2014).

After this peak around 2009, however, the percentage of HFT in total market activity began to decrease. Excessive competition and declining profits were likely the reasons for this decline. In many ways, HFT is a zero-sum game, and an increase in HFT firms tends to decrease each firm’s profits. According to the Tabb Group’s estimates, in 2012, the HFT industry earned a combined revenue of 1.8 billion dollars on US stock markets. This represents a decrease of roughly 70% from the 5.7 billion dollars earned in 2010 (Tabb, 2012).

It is also possible that the sudden drop in stock prices in May 2010—the so-called flash crash—contributed to reduced participation in HFT. Subsequently the HFT firm Eladian Partners was driven out of business in 2012. It was followed by Infinium Capital Management in 2014.

By 2014, the percentage of HFT as a proportion of the total value of all trades as estimated by Bogard had decreased to 48.5%. Since then, HFT’s share of trading value appears to have remained relatively stable. Because this proportion is close to 50%, it may be inferred that a situation exists in which each trade involves an HFT firm on one side and a non-HFT counterpart on the other. If this proportion were to exceed 50%, then the struggle between HFT firms on both sides of trades to achieve profits would lead to the elimination of some of them. A proportion of around 50% is thus regarded by some observers as the upper limit of sustainability for HFT. According to such analyses, the proportion of over 60% seen in 2009 is gone and never to return, as this would be unsustainable.

In Europe, HFT activities were much like those in the US, albeit with a lag of several years. The percentage of HFT as a proportion of equities trading in Europe based on the total value of all trades was 29% in 2009, and reached 38% in 2010 (World Federation of Exchanges, n.d.). Subsequently, however, it trended downwards, and is estimated to have sunk to 24% by 2014 (European Securities and Markets Authority, 2014).

3.2 HFT Firms Move to Japan from Saturated Markets Such as the US

The spread of HFT in Japan occurred later than in Europe or the US, where its share of market trading peaked around 2010. Its initial spread in Japan was driven by the launch of the Tokyo Stock Exchange “arrowhead” equities trading system in 2010 aimed at delivering high performance and ensuring reliability through world-class speed, reliability, and extendibility, all of which facilitated high-speed trading. The introduction of the arrowhead system paved the way for full-fledged HFT.

In 2010, just as the proportion of HFT in the US had begun declining from its peak, HFT activity began spreading rapidly in Japan. It is possible that the spread of HFT in Japan was also boosted by HFT firms shifting their activities to Japan from the saturated US markets, which were becoming less profitable.

In Analysis of High-Frequency Trading at Tokyo Stock Exchange (Hosaka, 2014), HFT represented 25.9% of the equity trading (value based) in Japan in 2014. This is roughly equivalent to the level in Europe at around the same time. However, it has been suggested that the proportion of HFT in Japan has grown since then, given the upgrade of the “arrowhead” equity trading system by the Japan Exchange Group in 2015,Footnote 3 and the quantum increases in trading speed and the number of transactions processed. Current levels of HFT in Japan, although lower than those in the US, are quite possibly higher than in Europe. It should be noted that the percentage of HFT as a proportion of equity trading in Australia was estimated to be 27% from January to March, 2015 (Australian Securities and Investments Commission, 2015). On the other hand, the proportions of HFT in two Asian markets, the Hong Kong and Singapore trading markets are thought to be very low (Wheatley, 2011) (Fig. 1).

Fig. 1
figure 1

Source Interim Report of ‘The Conference on the Impact of IT Innovation on Securities Markets’ (Fukuda, 2015). Notes (1) Comparisons are based on the total value of all trades. (2) Measurement periods are as follows. Japan: September 2012, January, and May 2013; US: January 2008–February 2010; Canada: August–November 2011; Australia: May–July 2012. All other countries: May 2013

International Comparison of the Proportion of HFT in Equity Markets.

Full size image

3.3 Activities of HFT Firms in Japan’s Highly Concentrated Market

Compared to markets in Europe and the US, Japanese equity markets are highly concentrated, meaning that the level of market fragmentation is low. This is reflected in the Tokyo Stock Exchange’s overwhelming share in equity trading.

As can be seen by the description of the nature of HFT trading in the US found in Flash Boys, the dispersion of trading over many different markets creates an environment that enables HFT firms to profit from arbitrage. Moreover, a large number of markets translates to a large number of opportunities for HFT firms to engage in market-making. In this sense, the greater the market fragmentation in a country, the more profit opportunities it provides for HFT, and the more attractive it is for HFT firms.

From this perspective, Japan, where equities trading is largely concentrated on the Tokyo Stock Exchange, may not at first glance appear to be an attractive market for overseas HFT firms. The fact that, as discussed below, foreign HFT firms are nevertheless highly active in Japan, is explained perhaps by the saturation of overseas markets, leaving Japan as a place where they can still survive and profit.

3.4 The Domination of the Japanese Market by Foreign HFT Firms

Japan introduced a registration system for HFT firms in April 2018. As of October 15, 2020, 55 HFT firms officially referred to as “those engaging in High Speed Trading” had been registered. With the exception of one Japanese firm, the head offices of all the registrants are located in countries other than Japan (see Table 1). The clear domination of Japan’s HFT by foreign players seems to indicate that HFT firms from the saturated US markets have now moved to Japan seeking profit opportunities.

Table 1 Location of the Head Offices of Registered HFT Firms in Japan (as of October 15, 2020)
Full size table

It is therefore quite likely that much of the profit from HFT in Japan is flowing out of the country. This situation may place domestic investors, particularly individual investors who are not engaged in HFT, at a disadvantage.

And yet, as of today, there does not appear to be much criticism of foreign domination of Japan’s HFT market in the country. Perhaps this is because there is a stronger awareness among market investors and other relevant parties of the positive contributions of HFT, such as supplying the market with liquidity. Or perhaps the paucity of criticism is due to a lack of awareness of the very existence of HFT firms among the general public in Japan.

Eventually, however, foreign HFT firms may one day be subject to closer scrutiny by the Japanese population. The situation is reminiscent of the time when some overseas investment funds, referred to as “vulture funds,” which invest in weak or even defaulting debt, beat down the prices of Japanese companies, resulting in a particularly cautious stance among the Japanese population.

4 Evaluation of the Impact of HFT on Financial Markets

4.1 HFT Effectively Enhances Market Functioning

Both the positive and negative effects of algorithmic trading, and of HFT in particular, have been the subject of debate from a variety of perspectives. First, considering the market-making algorithms used by the majority of HFT firms for trading, one of the most often cited positive effects of HFT is its role in supplying liquidity. Not only does the HFT activity of placing both buy and sell orders contribute to the supply of liquidity, but it is also instrumental in maintaining market stability.

Second, HFT trading based on arbitrage algorithms, another of the main forms of trading algorithms used by HFT firms, eliminates price divergence by rapidly placing orders whenever an arbitrage opportunity is discovered. To the extent that HFT eliminates these price differences, or market distortions, it can be viewed as improving market efficiency.

Therefore, given HFT’s impact on supplying capital, stabilizing markets, and enhancing market efficiency, it seems clear that HFT-style algorithmic trading effectively enhances overall market performance.

4.2 Empirical Research Overseas on the Effect of HFT on Market Performance

There is a large body of empirical evidence from outside Japan on how HFT supplies and enhances market liquidity. Table 2 shows some representative examples of this research.

Table 2 Empirical research on the supply of liquidity by HFT
Full size table

Empirical research from the US and Canada indicates that HFT effectively enhances market liquidity. Supporting these findings from a different perspective is an empirical analysis from the United Kingdom showing no evidence that increased HFT leads to higher trading costs for market participants by reducing market liquidity (Brogaard et al. 2014b).

Moreover, the conclusions drawn from significant empirical research conducted outside of Japan indicate that HFT has the effect of enhancing market efficiency. Table 3 shows some representative examples of this research.

Table 3 Empirical research on HFT’s enhancement of market efficiency
Full size table

The contribution of placing orders using HFT in determining efficient price levels has been confirmed through empirical analysis in the US and UK.

4.3 Results of Research in Japan

In Europe and the US, there is a significant body of academic research on HFT’s impact on markets. Much of this research presents a positive assessment of HFT’s impact on equity markets insofar as it contributes to enhancing the price discovery function and increasing liquidity.

The large amount of research on the subject in Japan, a relative newcomer to HFT, is notable for its focus on the changes in markets that occurred due to the advent of full-fledged HFT following the launch of the “arrowhead” trading system in 2010. For example, an analysis conducted by the Tokyo Stock Exchange (Hosaka, 2014) clarifies the characteristics of HFT firms by classifying orders into those placed by HFT firms and those placed by others, based on attributes of HFT as defined by the Committee on Economic and Monetary Affairs of the European Parliament (2012a) of having an order execution rate of less than 25%, and an order cancellation rate of more than 20%. According to Hosaka’s study: (1) few orders were placed in after-hours trading; (2) market orders were extremely rare; and (3) many orders were limit orders, placed outside the best bid and ask prices, which therefore tended not to be filled immediately, but rather to remained in the order book, unfilled, for a long period of time. This suggests that the orders placed by HFT firms provide the market with liquidity, and contribute to market stability.

Table 4 shows the results of some representative examples of empirical research into the impact of HFT on market liquidity in Japan.

Table 4 Empirical research on HFT’s enhancement of market efficiency
Full size table

4.4 The Possibility That HFT May Destabilize Markets

Despite extensive research about HFT throughout the world indicating that HFT effectively increases market liquidity and enhances market efficiency, there are others who argue that HFT-style algorithmic trading destabilizes financial markets. As already mentioned, the involvement of HFT is immediately suspected each time a “flash crash” occurs. Indeed, it is possible that in the event that markets become unstable for some reason, algorithms may act in unforeseen ways, resulting in an amplification of market instability.

Others have pointed out the possibility that a “runaway” algorithm, whether caused by a malfunction or some other factor, might cause disruption in markets. In addition, it is possible that, because HFT firms place, alter, and cancel orders swiftly and frequently, a few HFT firms may dominate price formation, with other investors unable to accurately grasp market conditions, thus resulting in price formation distortions. However, these effects remain in the realm of conjecture, as, in contrast to the abundance of research on the HFT’s positive effects on markets, there appears to be relatively little research on its negative effects. Nonetheless, it cannot conclusively be determined that the positive impact of HFT on markets outweighs its negative effects, as research on HFT’s negative effects may simply be more difficult to carry out due to data and technical limitations.

4.5 Conflicting Opinions on Whether HFT Amplifies Market Disruption

Even if it is now clear that HFT effectively enhances market functioning under normal, calm conditions, uncertainty remains regarding the value of HFT in times of crisis. It could well be that when markets become turbulent for some reason, HFT amplifies this turbulence. However, to date, this effect has not been supported by sufficient research.

In one example of such research regarding the relationship between HFT and the flash crash of 2010, Professor Andrei Kirilenko of MIT writes that when the flash crash occurred and price movements were accelerated, automated programmatic trading by HFT firms immediately withdrew the best bid and ask orders, which amplified price movements. He concludes that, when markets are under stress, biases in HFT order flows become more pronounced, leading to further price movements. In brief, HFT amplifies market disruptions (Financial Services Agency, 2016).

The above represents one view, although opinions about the potential harm caused by HTFs remain widely divided. Professor Terrence Hendershott of the University of California, Berkeley, writing that no meaningful evidence exists that the algorithms used in HFT vary their volume of trading according to changes in volatility, argues that algorithmic trading works to suppress rather than heighten volatility (Financial Services Agency, 2016).

4.6 The Issue of Fairness in Trading

Others claim that HFT, because of its ability to capture trading opportunities that might exist for only a moment, something which is difficult for average investors, creates what could be considered unfairness among investors. For example, even if average investors make decisions and submit orders based on the current market order book, by the time these orders reach the exchange, the order book will often have changed due to high-speed trading by HFT firms.

In addition, it is often pointed out that some HFT consists of market manipulation and other unfair trading practices. As stated, HFT entails the frequent placing, altering and canceling of large orders, and some have argued that these orders include some practices that are banned as market manipulation, such as “layering,” where traders place large orders that they have no intention of executing, and then cancel them when they are close to being filled. Indeed, there have been a few cases in Japan, where trading has been conducted with the intent to manipulate markets, and HFT firms have been exposed as the perpetrators.

5 HFT Regulation as a Preventive Measure

5.1 HFT Regulation and System Response in the US

Despite the extensive research on the possible impact of HFT, the underlying trading strategies used by HFT firms remain a black box; as a source of firm revenue, they are deliberately made difficult to discover. In response to a growing perception that HFT may lead to market disruption, and concern that some HFT firms may be involved in unfair trading such as market manipulation, despite there being no conclusive evidence of either, there is increasing public pressure to implement measures to prevent problems arising from HFT. These measures consist of the introduction of various rules by self-regulating bodies, and of regulation by authorities.

In Europe, regulators began by providing a clear definition of HFT for the purpose of regulation. By contrast, no such clear definition of HFT exists in the US. Even the definitions provided by the Commodity Futures Trading Commission (CFTC) and Securities and Exchange Commission (SEC) in 2010 are very general. Consequently, no regulation directly targeting HFT specifically has been introduced in the US. Still, the US is notable for its progressive application of regulations targeting some forms of HFT-style trading.

For example, the US bans naked trading, the practice of granting traders direct access to securities exchanges unfiltered by brokers’ order placement systems and without any intervening system to check customer orders. Where such a system exists, it is referred to as sponsored access. This ban on naked access substantially reduces excessive competition between securities companies to acquire HFT customers.

The use of stub quotes, which are limit orders that are deliberately set far lower or higher than the prevailing market price, is also banned in the US. They are used by market makers seeking to meet their price quote obligations without any intention of having their orders executed.

5.2 HFT Regulation in Europe

HFT is clearly defined in the EU through the Markets in Financial Instruments Directive II (MiFID II), a new, comprehensive regulative framework for financial and capital markets. Under this directive, all algorithmic trading firms, including HFT firms, have an obligation to report the details of their trading to the regulatory bodies. Additionally, securities exchanges are required to ascertain whether or not each order originates from an algorithmic trading firm.

Registration is also required for algorithmic trading firms that implement market-making strategies, and they must meet certain standards for supplying the market with liquidity.

5.3 HFT Regulation in Japan

In Japan, government and cabinet office ordinances regulating HFT came into force in April 2018, pursuant to amendments to the Financial Instruments and Exchange Act. HFT firms (“those engaging in High-Speed Trading”) are now required to register and provide prior notification of their trading strategies, and registration will be rejected if there are any shortcomings in the firm’s equipment or systems. As already noted, 55 HFT firms were registered as of October 2020.

A definition of HFT firms has been established in Japan, although it is not as clear as the definition established in Europe. In Japan, HFT is commonly understood as trading where methods are implemented to transmit orders, etc. in a shorter time than usual, and mechanisms are established to prevent competition with other ordinary orders.Footnote 4 In addition, HFT firms have an obligation to prepare and preserve trading records. The supervisory authority can demand and inspects reports, and issue business improvement orders.

Regulation was not introduced in Japan with the intention of eliminating HFT firms on the basis of any inherent impropriety on their part. Rather, Japanese regulation aims to enable regulators and securities exchanges to obtain an accurate understanding of the actual status of HFT firms, which would otherwise be unclear, and to promote the establishment of an environment for enhancing their supervision.

The registration system was introduced because it was judged, with reasonable grounds, to be necessary for authorities to grasp the de facto situation regarding HFT. Many HFT firms are unlisted and disclose little information publicly. This makes it difficult for authorities to gain an understanding of their actual status and activities. Without the registration system, it would be necessary for the Tokyo Stock Exchange and other private sector companies to monitor the situation autonomously. This would entail significant cost, and certain aspects that could be difficult to implement. By introducing a registration system, Japan has clearly indicated its position, with the national administration responsible and paying for the system, and taking measures when any unfairness is exposed. There were some initial concerns that the introduction of a registration system would inhibit HFT activities, but at present, there is no evidence to support these concerns.

5.4 Few Cases of HFT Unfair Trading Have Been Exposed in Japan

To date, very few cases of unfair trading related to HFT have been exposed in Japan. Three reasons can be suggested for this.

First, markets in Japan are not as fragmented as those in the US. As a result, there is relatively little market distortion, and far less HFT in Japan than in the US or Europe. In addition, even if unfair trading by HFT firms is discovered, some aspects of exposing such trading may be difficult due to the limitations placed on regulatory controls in Japan. Finally, it is possible that regulators have not been able to trace unfair trading by HFT firms due to inadequate technology.

Of these reasons, the last seems the most likely to have affected HFT regulation in the past. Indeed, it seems that it was technically difficult for regulators to detect unfair trading by HFT firms due to the extremely short timeframes involved.

With the introduction of a registration system however, regulators’ grasp and assessment of unfair trading are becoming increasingly more effective. Moreover, private-sector initiatives are also helping enhance monitoring functions through the application of machine learning to vast quantities of market data using AI technologies, and these are becoming more adept at discovering suspicious activity.

Stronger relationships between the private sector and regulators, including the broad supply of information to regulators by private sector companies, should contribute to suppressing unfair trading.

According to the Japanese Financial Services Agency, “In contrast with Europe and the US, the amount of trading in Japan that unfairly exploits market fragmentation, etc. is limited. Even so, there have been cases of market manipulation using algorithmic trading, or working with algorithms, where corrective action has been required.” It goes on to describe cases where monetary penalties have been imposed in cases involving market manipulation activities in which the offenders placed trading orders that they never intended to execute (Financial Services Agency, 2016).

6 HFT and the Securities Sector in Japan Today

6.1 Japanese Securities Companies Delayed the Introduction of Practices from the US

Finally, I would like to focus on two recent trends in the activities of HFT firms in Japan. In both, HFT firms are thought to benefit in some way by obtaining information on stock orders placed by investors. These cases have once again ignited the smoldering debate in academia on whether, after all, HFT benefits or damages the interests of other investors.Footnote 5 The mechanism behind both these HFT activities was imported from the US. In this sense, Japan, a relative newcomer to HFT, is following in the footsteps of the US model.

The first trend concerns smart order routing (SOR), a common practice in the securities business in the US. Securities companies have an obligation to execute orders received from their customers at the best terms possible, based on publicly available information on bid–ask quotes and trades, after considering factors such as prices, costs, speed, and the possibility of order execution. This is referred to as their duty of best execution. SOR is an automated system aimed at helping securities companies fulfill this duty of best execution by applying an algorithm to instantaneously select the market offering the best price.

6.2 Smart Order Routing and Order Book Information

According to a report in The Nikkei in November 2019 (“Japan’s Flash Boys”), published by Ken Kawasaki (2019), an online securities broker working under the umbrella of one of Japan’s financial groups, received orders from customers, many of whom were individual investors. He placed these orders on the optimal market, and used SOR between the Tokyo Stock Exchange and the financial group’s own proprietary trading system (PTS). It should be noted that, in Japan, the obligation to trade stocks only through exchanges was abolished in 1998, and the ban on the proprietary trading system operation was lifted as a result.

After receiving a customer order, the online securities broker in question sent it first to the PTS, and then to the Tokyo Stock Exchange, if this was judged to be the optimal market. Even if the order eventually wound up on the Tokyo Stock Exchange, it would be exposed for a certain period of time on the PTS order book. This time was around 0.1–0.3 s. This may seem like only an instant, but for HFT firms, the interval of 0.1–0.3 s is an extremely long time.

It appears that HFT firms were able to obtain information on these customer orders, and then swiftly place orders on the Tokyo Stock Exchange in anticipation of these orders arriving. When this happened, the HFT firm that anticipated the order may have been able to profit from the trade, and the individual investor whose order was anticipated may have been forced to trade at a less favorable price as a result. According to The Nikkei, this scheme was introduced in October 2019.

6.3 The Emergence of Japan’s Flash Boys?

The scheme described above closely resembles that described in Michael Lewis’ Flash Boys, where an HFT firm obtained information on orders placed by other investors from the order book, and profited by instantly placing, altering, and canceling orders accordingly (Lewis, 2014). Their strategy was analogous to cheating at rock-paper-scissors, waiting to see their opponent’s move, then playing their hand an instant later. The scheme described above is sometimes referred to as Japan’s Flash Boys.

The aim of temporarily exposing customer orders on a PTS is sometimes explained as an attempt to stimulate counter orders, thus enhancing trading activity and improving execution rates. The Japanese financial group concerned revised its SOR execution method in November 2019, perhaps in view of criticism from some quarters, to prevent information on customer orders being temporarily visible from the outside.

Fig. 2
figure 2

Nomura Research Institute, from information published in The Nikkei newspaper

Are HFT Firms Anticipating Orders by Individual Investors?

Full size image

However, it has been pointed out that even after this revision, in the case of customer orders that are sent by SOR to the PTS but which cannot be executed there and are thus transferred to the Tokyo Stock Exchange, there is still room for HFT firms to anticipate and profit from the arrival of these unexecuted orders. Figure 2 shows SOR and a flow of individual investor orders to HFT firms in Japan.

6.4 The Movement to Introduce Payment for Order Flow in Japan

The other trend that I would like to focus on is the spread of payment for order flow (PFOF) to Japan, a practice common among securities companies in the US. This refers to a scheme whereby a securities company passes customer orders (transaction rights) on to market makers such as HFT firms, and receives a rebate (compensation) in return.

The setting for this scheme is, of course, the security company’s PTS. HFT firms pay commissions to the PTS, and the PTS pays rebates to the online securities broker. In other words, rebates flow indirectly from HFT firms to the online securities broker via the PTS.

It is thought that HFT firms are willing to pay rebates for information on orders issued by individual investors because this allows them to enhance the precision of their proprietary algorithmic trading by analyzing this big data using AI, and utilizing it for purposes such as predicting the trading trends of individual investors in Japan.

Figure 3 shows how Japanese online securities brokers receive rebates from HFT firms under PFOF practices.

Fig. 3
figure 3

Nomura Research Institute, from information published in The Nikkei newspaper

Spread in the Receipt of Rebates from HFT Firms.

Full size image

6.5 Information on Orders by Individual Investors is Valuable for HFT Firms

It may seem that, unlike orders by large investors, which can cause significant movements in the market, the small-scale orders placed by individual investors provide HFT firms with few profit opportunities. However, the accumulation of many of these small-scale orders by individual investors can have a substantial impact on the market. Moreover, large orders by institutional investors are sent to the market after being split into small portions by securities companies, to prevent them from affecting market prices, or even to prevent them from being detected by other market players. By analyzing orders by individual investors, HFT firms may well be able to enhance the precision with which they can differentiate between small-scale orders and large orders that have been split into several portions. If the presence of a large-scale order is detected, then they will be able to anticipate the arrival of later portions of the split order on the market, thus achieving significant profits.

It is reasonable to suggest that for these reasons, information on orders placed by individual investors is valuable for HFT firms, and they are thus willing to pay fees to obtain it.

6.6 Against the Backdrop of Commission-Free Trading

In this way, the movement by Japanese online securities brokers to introduce the US-style practice of PFOF undoubtedly represents an effort to secure new sources of revenue. In recent years, there has been a clear trend towards lower trade commissions (transaction fees) for share trading around the world. Japan is no exception.

Securities companies require other sources of income to supplement the reduction in revenue from lower commissions. Generally speaking, these consist of sources such as interest revenue from money lent to customers for margin trading, and stock loan fees changed for lending customers’ shares to third-party investors wishing to a take a short position in that stock.

In Japan, however, with its extremely low interest rates, interest revenue from margin trading and stock loan fees from lending shares have both sunk to very low levels. It was in this context that Japan’s online securities brokers began to seek to secure a new source of revenue through the introduction of PFOF. In addition, it is also possible that Japan’s online securities brokers may be shifting their business models progressively to resemble more that of US Robinhood-style operations. Robinhood, an online (app-based) securities broker in the US that offers almost entirely commission-free trading, passes almost all the orders that it receives from customers to HFT firms. It is estimated that, as of early 2018, it derived more than 40% of its revenue from rebates from HFT firms.

6.7 Are the Interests of Individual Investors Being Protected?

Under PFOF systems, securities companies provide HFT firms with big data on orders, most of which have been submitted by individual investors, and receive rebates in return. These rebates are used by securities companies to fund the provision of commission-free trading platforms to individual investors. This scheme closely resembles the business models used by digital platformers, which provide users with free online services funded by external income from targeted advertising, etc. utilizing personal data acquired through these online services.

In this way, individual investors are effectively providing their order data to HFT firms in return for lower, or zero, trade commissions. It is possible however, that through this exchange, individual investors are being driven into a more disadvantageous trading environment by HFT firms. From this perspective, it is still not entirely clear whether, in fact, individual investors are receiving equivalent value in return. Further verification of this point is necessary in the future.

Japan’s securities companies, operating under a persistent low interest rate regime, have a weaker earnings base than their US counterparts. For online securities brokers in particular, the importance of rebates from HFT firms may eventually be even more important than in the US. If this arrangement becomes institutionalized, then HFT firms active in Japan would play an even more important role than those in the US or elsewhere in supporting business models in the securities industry. Securities companies and HFT firms would be mutually dependent, bound together by a shared fate.

6.8 Research on HFT is Still in Its Infancy

As shown above, despite a relatively clear consensus on the contribution made by HFT firms to enhancing market efficiency, it is still undetermined whether HFT amplifies market disruptions, and whether it damages the interests of other investors, including individual investors. This uncertainty is no doubt due partly to a lack of clarity regarding the actual nature of HFT, conducted at speeds and frequencies that defy human comprehension. For both regulators and academics, research into HFT is still in its infancy.

As research on the subject advances however, and the merits and deficiencies of HFT become clearer, perhaps HFT will be able to evolve into a factor that contributes to further market development and new business models for the securities industry, such as those described in the last section. In this context, further developments in HFT research are to be welcomed.