Algorithm and architecture design of the motion estimation for the H.265/HEVC 4K-UHD encoder
- First Online:
- Cite this article as:
- Pastuszak, G. & Trochimiuk, M. J Real-Time Image Proc (2016) 12: 517. doi:10.1007/s11554-015-0516-4
- 1.8k Downloads
This paper presents the algorithm and the architecture of the high-throughput motion estimation system for the H.265/HEVC encoder. The design allows the processing of 2160p@30fps videos at the clock frequency of 400 MHz. The architecture embeds two parallel processing paths for the integer-pel and the fractional-pel motion estimation. The paths share the same memories. Access conflicts are avoided by the use of dual-port modules and register buffers for reused samples. In each clock cycle, the integer-pel and the fractional-pel path can evaluate one and four motion vectors for an 8 × 8 luma block, respectively. A separate interpolator for chroma additionally increases the throughput. The integer-pel path supports test zone search for 8 × 8 prediction blocks. The motion estimation for larger blocks is performed by the utilization of results of the 8 × 8 search. The search for rectangular PUs is performed only at the fractional-pel level and reuses partial costs computed for square PUs. As a consequence, a significant amount of computation is saved. Synthesis results show that the design can operate at 200 and 400 MHz when implemented in FPGA Arria II and TSMC 90 nm, respectively. The implemented algorithm is verified in the HM16 software. If 2160p@30fps videos are encoded with the low-delay configuration, BD-PSNR and BD-rate are equal to −0.026 dB and 1.64 %, respectively.
KeywordsVideo codingMotion estimationInterpolationH.265/HEVCFPGAVery large-scale integration (VLSI)
Research and standardization efforts in video coding led to the specification of the H.265/HEVC standard [1, 2] in 2013. At the same quality of the reconstructed video, the standard provides an improvement in compression efficiency of about 35–50 % compared to its predecessor H.264/AVC . However, the better compression efficiency is achieved at the price of increased computational complexity. Although the general structure of the encoder and the decoder remains the same, there are many changes in the algorithm. Instead of 16 × 16-pixel macroblocks, the new standard applies coding tree units (CTUs), which can be up to 64 × 64 pixels in size. Each CTU can be recursively split into square coding units (CUs) with the minimal size of 8 × 8 pixels. Each 2N × 2N CU can be partitioned into predictions units (PUs). N can be equal to 4, 8, 16, or 32. There are eight allowable partition shapes: two square shapes (2N × 2N and N × N), two symmetric rectangular shapes (N × 2N and 2N × N), and four asymmetric rectangular shapes (2N × 3N/2, 2N × N/2, 3N/2 × 2N, and N/2 × 2N). Each inter PU has a separate motion vector (MV). Similar to H.264/AVC, the H.265/HEVC allows quarter pixel accuracy MVs. There are new interpolation schemes to compute fractional-pel positions. In particular, 7-tap and 8-tap filters are used for the luma interpolation of half-pel and quarter-pel positions, respectively. Chroma samples are computed using 4-tap filters. Although design and implementation of digital filters is a thoroughly explored issue, high-throughput video encoders require some effort to obtain efficient hardware solutions.
With the exception of our previous designs [4, 5], architectures for the motion estimation (ME) consist of two parts assigned to the integer-pel and fractional-pel search [6–15]. This approach requires separate reference pixel buffers for each part. The integer-pel part usually applies the hierarchical strategies to extend the search range, which involves quality losses. Most architectures use non-adaptive search patterns and their resource consumption is large [6–10]. The architecture supporting Multipoint Diamond Search proposed in  requires less resource; however, it only supports 16 × 16 blocks, limiting the compression efficiency.
Some high-throughput interpolators have been proposed in literature for H.264/AVC [5–9]. Their scheduling assumes two successive steps, one for the half-pel interpolation and another for the quarter-pel interpolation. This approach is natural in terms of the specification of quarter-pel computations which refer to results of half-pel computations. This dataflow cannot be applied directly in H.265/HEVC since quarter-pel samples are computed using separate filters. In particular, more filters are needed in the second step. Furthermore, the hardware cost increases due to a larger number of filter taps and much higher throughputs required (more partitioning modes). Some interpolator architectures designed for H.265/HEVC have been described in literature [12–15]. They achieve throughputs suitable for video resolutions from 1080p to 4320p. All the designs neglect the interpolation for merge modes. Three designs [12–14] are based on the assumption that the size of prediction units is selected at the integer-pel ME. If the processing of more sizes has to be performed, the throughput is decreased accordingly. One design  supports three prediction block sizes (16 × 16, 32 × 32, and 64 × 64); however, it consumes a large amount of hardware resources. Generally, a compression-efficient and high-throughput implementation requires more hardware resources and increases power consumption. Therefore, there is a need for solutions in which these parameters are optimized.
This study presents the high-throughput ME architecture dedicated to the H.265/HEVC encoder. Similar to the previous works [4, 5], the architecture can check one integer-pel motion vector for an 8 × 8 block in each clock cycle. As an arbitral order of motion vectors is allowed, the architecture supports the test zone search (TZS) algorithm used in the HM software. A significant amount of computation is saved for 16 × 16 and 32 × 32 prediction blocks by the exploitation of results of the 8 × 8-block search. In the case of rectangular PUs, only MVs checked for the fractional-pel ME of 2N × 2N PUs are evaluated, which additionally reduces the complexity at small quality losses.
The present study has four novel contributions at the architecture level. Firstly, the use of dual-port memories and register buffers for reused data allows shared and conflict-free access from two processing paths corresponding to the integer-pel and the fractional-pel search. Secondly, the extension of the interpolation to 9 × 9 blocks allows the evaluation of four 8 × 8 fractional-pel blocks at a small increase in the resource cost. Thirdly, the architecture enables the two-dimensional continuous interpolation of 9 × 9 blocks with reconfigurable and dedicated filter cores. Fourthly, the separate chroma interpolator additionally increases the throughput and design flexibility.
The rest of the paper are organized as follows: Sect. 2 reviews previous developments on the hardware design of the adaptive motion estimation. Section 3 describes the new architecture of the H.265/HEVC motion estimation system. The applied scheduling is described in Sect. 4. Section 5 presents the motion estimation algorithm executed by the proposed architecture. Section 6 provides implementation results. Finally, the paper is concluded in Sect. 7.
2 Design for adaptive motion estimation
The adaptive computationally scalable motion estimation algorithm allows video encoders to achieve close to optimal efficiencies in real-time conditions . The algorithm can employ different search strategies to adapt to local motion activity, and the number of checked search points is set by the encoder controller for each macroblock. The algorithm can achieve results close to optimum even if the number of search points assigned to macroblocks is strongly limited and varies with time.
In order to process blocks of 8 × 8 samples, the interpolator embeds 64 reconfigurable filters . The reconfiguration allows the computation of four fractional-pel positions (e.g., 0, 1/4, 1/2, and 3/4) for both luma and chroma samples. The number of filters corresponds to the size of blocks processed in the main path. Although the interpolation parallelism is high, the throughput is limited by the reading of 8 × 8 blocks at the input. In particular, two and four 8 × 8 blocks must be read to obtain the 1D and 2D interpolation of three fractional-pel positions for one block, respectively. More clock cycles are utilized when the interpolation is performed in two dimensions. If 100 cycles are available for each 8 × 8-pixel block (2160p@30fps), the interpolation around two integer-pel MVs can be performed for luma. Particularly, one 1D luma interpolation with the cross pattern takes 16 cycles, whereas 2D interpolation for nine positions takes 27 cycles. Two corresponding chroma blocks are interpolated in 10 cycles for one position. Totally, 96 out of 100 cycles are utilized for the luma and chroma interpolation. 54 cycles are available for the integer-pel search interleaved with memory reads for the fractional-pel estimation. Although the throughput is significantly improved compared to other designs [12–14], it is still insufficient to evaluate the greater number of PU sizes. Additional interpolations are indispensable to support merge modes.
3 New architecture
In the architecture described in the previous section , the integer-pel and the fractional-pel search share the same processing path with the interleaved processing. As a consequence, the number of clock cycles assigned to the integer-pel estimation is decreased almost by half, which has a negative impact on the compression efficiency. The main bottleneck is introduced by the memory read port able to provide one 8 × 8 block in each clock cycle. In order to resolve the problem, the new architecture incorporates dual-port memory modules instead of two-port ones. The main advantage of dual ports is that they can operate in either the read or the write mode. In the architecture, the first port is assigned to the integer-pel path, whereas the second is used as the input to the interpolator. The interpolator incorporates the register buffer at the inputs stage to reuse samples from the second path. Since the interpolator does not read data in each clock cycle, some cycles can still be utilized to write the reference pixels for the following CTUs. The same approach is applied to the memory storing original samples.
9 × 9 blocks released from the interpolator must be written to a buffer to wait for the end of the preselection process. Some predictions corresponding to preselected MV candidates should be kept until they are forwarded to the reconstruction loop and the rate-distortion optimization. The buffer is outside the ME system and will be used to integrate with other encoder modules .
The horizontal interpolator computes the 9 × 16 sample array in four clock cycles and then forwards it to the vertical stage. The vertical interpolator can be implemented as the 9 × 9 array of reconfigurable filters which determine a 9 × 9 block for one fractional-pel MV in each clock cycle. However, the hardware cost of 81 reconfigurable filters is significant. To save resources, the vertical stage incorporates 54 dedicated and nine reconfigurable filters. Each of three fractional-pel interpolations (1/4, 1/2, and 3/4) is performed with 18 dedicated filters. Separate bypass paths transfer 18 samples not interpolated vertically. Each bypass path includes the rounding adder and the range limiter. Nine reconfigurable filters perform all interpolations for the most right column in three cycles. The fourth cycle is utilized to transfer nine samples through the bypass path. The remaining eight columns are horizontally rotated between registers feeding dedicated filters and the bypass path. In particular, the register content is moved by two columns in each clock cycle. Each register column is assigned to one of the three groups of dedicated filters or to the bypass path. As a consequence, the 9 × 9 blocks released from the vertical stage consists of samples interpolated for four fractional-pel MVs. Thus, SADs must be accumulated in parallel for 16 fractional-pel MVs in four clock cycles. One multiplexer at the interpolator output is used to restore locations of four 2 × 9 sample groups. Another multiplexer vertically transposes positions in the most right column if the result for the 3/4 interpolation is released.
The design of reconfigurable filters is well suited to FPGA devices since multiplexers are embedded in the same logic cell as the following adder/subtractor. The luma and chroma filter cores embed 12 and 10 adders/subtractors, respectively. The previous architecture required 22 adders/subtractors for the filter supporting both luma and chroma and 17 for luma. Therefore, the significant reduction of resources is achieved when the filter is limited to the luma processing.
Figures 10 and 11 depict architectures of dedicated filters used at the vertical stage for the half-pel and quarter-pel interpolation, respectively. The half-pel filter embeds 10 adders/subtractors whereas the quarter-pel filter consumes one more. Dedicated filters embed the rounding adder in the tree. The output multiplexer accomplishes the clipping (CLIP) of the final result to avoid overflow and underflow.
The fractional-pel ME needs 16 clock cycles to evaluate 64 MVs around one integer-pel MV. Thus, the search can be performed around six MVs for a given 8 × 8 block. However, some cycles are required to interpolate MVs identified for the merge mode, in particular, four cycles are utilized to obtain the 8 × 8 interpolation for one MV. If the merge MV falls in the range of the regular fractional ME, no additional cycles are required. It is assumed that 48 cycles are allocated to regular fractional ME around three integer-pel MVs (8 × 8, 16 × 16, and 32 × 32 PUs). The remaining 52 cycles are utilized to process 13 merge mode candidates determined for different CU divisions. The regular factional-pel ME for a given PU is skipped if its range matches that for a larger PU. Saved cycles are utilized to evaluate more merge mode MVs. Since the availability of most of merge MVs depends on the mode decision for preceding CUs/PUs, merge mode candidates are evaluated at the same stage as the reconstruction loop and the CU/PU mode decision.
Interpolation filters specified in H.265/HEVC refer up to eight luma samples located in row/column at neighboring pixel positions. Therefore, the 2D interpolation of one sample requires access to the 8 × 8 reference block. If four blocks are accessed, the output can be extended to the 9 × 9 block. Provided that 8 × 8 blocks appear at the interpolator input, four cycles are taken to load the input registers. The location of the blocks can be identified by specific MVs, as shown in Fig. 5. For convenience, the following description will refer to motion vector differences (MVDs) relative to the integer-pel position around which the fractional-pel search is executed. If two horizontally adjacent 8 × 8 blocks are obtained for MVDs equal to (−4, 0) and (4, 0), the interpolator can compute MVDs equal to (1/4, 0), (1/2, 0), (3/4, 0), (−1/4, 0), (−1/2, 0), and (−3/4, 0). The same rule applies to the vertical processing. Four reference blocks required for the 2D interpolation have the following MVD: (−4, −4), (4, −4), (−4, 4), and (4, 4).
To perform the luma interpolation, four 8 × 8 reference blocks are taken from the input and written to the first ring buffer. The buffer consists of four register groups (FRB–FRB ), each of which keeps four 16 sample rows. In each clock cycle, the rows are vertically rotated between register groups. Row indices are indicated in Fig. 14. Each reference block is simultaneously written to two register groups. Since each row is composed of samples taken from two reference blocks, two groups are half-filled with new samples in one cycle. Due to the rotation, the first/third block is written to FRB and FRB , whereas the second/fourth block is written to FRB  and FRB . If the 3/4 interpolation is performed, samples written to FRB  registers are horizontally transposed. The FRB  registers feed horizontal filters. The filtering result is obtained with the delay of two clock cycles. Horizontally interpolated samples corresponding to four rows are written to horizontal registers (HR) in each clock cycle. Every fourth clock cycle, 12 rows kept in HR and four rows available at filter outputs are forwarded to the second ring buffer (SRB). The buffer feeds 63 vertical filters and 18 bypass paths. The SRB is composed of nine columns. Eight of them are horizontally rotated by two positions in each clock cycle. Each two of six columns are assigned to a group of 18 dedicated filters supporting one particular type of the interpolation (either 1/2, 1/4, or 3/4). Two columns are assigned to 18 bypass paths. Similar to the horizontal stage, the filtering result is obtained with the delay of two clock cycles. The rotation in the second ring buffer allows the processing of eight columns with each filter type. On the other hand, multiplexers are required at outputs to restore appropriate locations of columns in the 9 × 9 block. One of nine columns is not rotated, and it feeds nine reconfigurable filters. The filters are reconfigured in each clock cycle to support one particular type of the interpolation. For the 3/4 interpolation, samples kept in SRB  are vertically transposed.
5 Search strategy
The proposed ME architecture can check an 8 × 8 prediction for one integer-pel MV and four fractional-pel MVs in each clock cycle. In practice, the number of evaluated MVs is limited and depends on the clock frequency and the video resolution. If the motion estimation operates at the frequency of 400 MHz and processes 2160p@30fps videos, the number of integer-pel MVs per each 8 × 8 block in the original image is about 100. This number should be allocated to all evaluated PUs corresponding to the block. Taking into account wider search ranges required for the 2160p@30fps resolution, numbers of MVs allocated to particular PUs can be too small to achieve a high compression efficiency. In the case of the fractional-pel ME, the available number of clock cycles can also limit the efficiency. Other limitations stem from the encoder dataflow, which introduces the delay between the ME and the final mode decision (based on the rate-distortion optimization). The delay causes some MV predictions to be unknown at the ME. Thus, costs of evaluated MVs cannot be estimated reliably. Moreover, the determination of predictions for merge modes must follow the mode decision for preceding blocks.
The search range is set to (−64, 63) × (−64, 63).
Test zone search is performed only for 8 × 8 PUs. It is interrupted when the number of checked MVs achieves the limit specified for a given resolution. The limit corresponds to the number of clock cycles assigned in the hardware architecture (e.g., 92 for 2160p@30fps). If some 8 × 8 PUs within the 32 × 32 unit do not utilize all allowable cycles, the remaining cycles are added to continue the interrupted search. This reallocation makes losses in the compression efficiency negligible.
The integer-pel motion estimation for 16 × 16 PUs is performed by utilizing results from the 8 × 8 search. Four MV candidates are taken from MVs found for 8 × 8 blocks included in a given PU.
The integer-pel motion estimation for 32 × 32 PUs is performed by utilizing results from the 16 × 16 search. MV candidates are determined according to the rule applied in the 16 × 16 search.
Rectangular PUs are evaluated within the range of the fractional-pel estimation corresponding 2N × 2N PUs. Although this simplification significantly reduces the ME complexity, it has a small impact on the average compression efficiency (0.3 %).
MV costs are estimated based on results of the 8 × 8 search if a neighbor belongs to the same CTU. In this case, MV differences are computed with the assumption that neighbors are 8 × 8 blocks. In the remaining cases, actual MV predictors are taken from adjacent CTUs.
Only merge mode candidates are evaluated for 64 × 64 PUs and their rectangular partitions. The exclusion of the 64 × 64 search decreases the compression efficiency by 0.8 % (−0.02 dB), on average.
At least three merge mode candidates are evaluated for each PU if the video resolution is 2160p@30fps. More candidates can be processed if any of the three following conditions are true: First, merge MVs fall in the range of the fractional-pel search for the same or a larger PU. Second, fractional-pel search for a given PU matches that for a larger PU. Third, the resolution is lower than 2160p@30fps. The conditions stem from the scheduling and allow a better utilization of available clock cycles. In particular, more merge modes are evaluated to avoid the redundant processing and/or no-operation cycles.
The final MV is not selected with sum of absolute transformed differences (SATD) used in the HM software. Instead, candidate MVs are selected based on SAD at the fractional-pel stage. Four candidates are selected for square PUs. The remaining (rectangular) PUs have one candidate MV. It is assumed that corresponding predictions are used in the mode selection based on the rate-distortion analysis. This approach decreases the compression efficiency by 0.3 % compared to the use of SATD.
The reuse of results of the 8 × 8 search saves a significant amount of computations. Particularly, eight integer-pel MVs are evaluated for larger PUs including a given 8 × 8 block. Moreover, MVs for the larger PUs are reused for smaller ones.
Evaluation results for BD-PSNR and BD-Rate
2560 × 1600
1920 × 1080
832 × 480
416 × 240
1280 × 720
3840 × 2160
6 Implementation results
Resource consumption for FPGA and ASIC technologies
Arria II GX (ALUT)
TSMC 90 nm (gate)
3598 (7.71 %)
27,836 (6.59 %)
3412 (7.32 %)
28,982 (6.86 %)
24,202 (51.89 %)
240,072 (56.80 %)
12,143 (26.03 %)
97,124 (22.98 %)
546 (1.17 %)
3264 (0.77 %)
2742 (5.88 %)
25,386 (6.00 %)
For the ASIC technology, the design can operate at the frequency of 400 MHz. This performance enables the encoder to allocate about 100 clock cycles per each 8 × 8 block if the resolution is 2160p@30fps. The estimated power consumption of the ASIC implementation is equal to 293 mW. The high power consumption is caused by memories keeping reference and original pixels. The FPGA implementation can operate at 200 MHz. As a consequence, the throughput is decreased by half.
The luma and chroma paths incorporate 64 dual-port and 16 two-port memory modules, respectively. The modules store reference pixels. Each module in the luma path is 0.75 kB in size. In the case of the chroma path, the size is 1.5 kB. The joint capacity of 72 kB allows the search range of (–64, 63) × (−64, 63) for both luma and chroma. Wider ranges are possible at the cost of the increased memory size. The original luma samples are stored in a separate dual-port memory with a capacity of 4 kB. This capacity is sufficient to keep samples for one CTU. Since the ME system is pipelined based on 32 × 32 units, the assignment of memory subspaces is swapped between four processing stages (the writing, the integer-pel ME, the fractional-pel ME, and the merge mode evaluation).
Byun et al.  presented the H.265/HEVC integer-pel full search architecture supporting all prediction unit sizes with the range of (−32, 31) × (−32, 31). The design consumes 3.56 M gates and 23 kB memories. The hardware cost of the motion estimation system described in this paper is much smaller (422.7 k gates and 76 kB memories). Moreover, the search range is wider [(−64, 63) × (−64, 63)]. The low-power integer-pel design was proposed by Sanchez et al. . Its resource consumption is relatively low (50 k gates and 82 kbit memories). However, it supports only 16 × 16 blocks and a narrow search range, which does not exploit the compression potential of H.265/HEVC.
Comparison with other FPGA architectures
Arria II GX
Arria II GX
4077 + 16547
1000 × parallelism/resources
Dynamic power (mW)
Luma and chroma
Luma and chroma
Comparison with other ASIC architectures
16 × 12 (2D)
1000 × parallelism/resources
Luma and chroma
Luma and chroma
Luma and chroma
Most referenced designs support only the luma interpolation [12, 14, 15]. The FPGA implementation proposed by Afonso et al.  achieves a high frequency due to deep pipelining and the better device. The proposed architecture can also be modified to operate at higher frequencies by the insertion of registers. This modification would not increase the logic resources since at least one flip-flop is embedded in each ALUT. However, the power consumption would be increased. Moreover, the gain in the frequency would not compensate the increased latency of the deeply pipelined processing path composed of the luma predictor, the interpolator, and the cost estimator. The latency of the path affects timing constraints corresponding to the final mode decision and the availability of corresponding MVs. Thus, it would be difficult to determine merge mode candidates and MV costs for the highest throughput.
Although the hardware cost of the interpolator is decreased compared to the previous one , the proposed ME system is more complex. Particularly, the compensator in the previous architecture consumes 42.5 k gates, whereas the inter luma/chroma predictor and the cost estimator in the new one require 129.4 k gates. There are two main reasons of the increase. First, separate processing paths for the integer-pel and the fractional-pel are used. Second, four costs are simultaneously evaluated in the fractional-pel path. Since most logic resources are contributed by interpolators (265.5 k gates), the increased complexity in the remaining modules is relatively small in terms of the whole ME system. The throughput is increased by the factor of 1.85 (100/54) and 3.1 (100/32) for the integer-pel and fractional-pel processing, respectively.
The ME architecture is developed for the H.265/HEVC encoder. The design embeds two parallel processing paths for the integer-pel and the fractional-pel motion estimation. The paths share the same dual-port memories. Internal buffers and the scheduling allow the writing of reference samples through the port assigned to the fractional-pel path. The architecture supports TZS for 8 × 8 prediction blocks. The motion estimation for larger blocks is performed by utilizing results of the 8 × 8 search. The search for rectangular PUs is performed only at the fractional-pel level and reuses partial costs computed for 2N × 2N PUs. The design achieves the best ratio of the throughput to hardware resources compared to other designs. The design can check about 100 integer-pel MVs for each 8 × 8 input block when encoding 2160p@30fps video at the 400 MHz. Within future works, the proposed ME system will be integrated with the intra encoder  to support inter modes.
This research was supported in part by PL-Grid Infrastructure.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.