camera_metadata: update high speed video recording API

Update high speed video recording API to support batch mode recording.

Bug: 21442271
Change-Id: I62de6067c75fcfbe6320dd987cce65843d810021

2 files changed, 139 insertions, 18 deletions

diff --git a/core/java/android/hardware/camera2/CameraCharacteristics.java b/core/java/android/hardware/camera2/CameraCharacteristics.java
index d5867a9..85e8827 100644
--- a/core/java/android/hardware/camera2/CameraCharacteristics.java
+++ b/core/java/android/hardware/camera2/CameraCharacteristics.java
@@ -626,35 +626,54 @@ public final class CameraCharacteristics extends CameraMetadata<CameraCharacteri
             new Key<Integer>("android.control.maxRegionsAf", int.class);
 
     /**
-     * <p>List of available high speed video size and fps range configurations
-     * supported by the camera device, in the format of (width, height, fps_min, fps_max).</p>
-     * <p>When HIGH_SPEED_VIDEO is supported in {@link CameraCharacteristics#CONTROL_AVAILABLE_SCENE_MODES android.control.availableSceneModes}, this metadata
-     * will list the supported high speed video size and fps range configurations. All the sizes
-     * listed in this configuration will be a subset of the sizes reported by {@link android.hardware.camera2.params.StreamConfigurationMap#getOutputSizes } for processed
-     * non-stalling formats.</p>
-     * <p>For the high speed video use case, where the application will set
-     * {@link CaptureRequest#CONTROL_SCENE_MODE android.control.sceneMode} to HIGH_SPEED_VIDEO in capture requests, the application must
+     * <p>List of available high speed video size, fps range and max batch size configurations
+     * supported by the camera device, in the format of (width, height, fps_min, fps_max, batch_size_max).</p>
+     * <p>When CONSTRAINED_HIGH_SPEED_VIDEO is supported in android.control.availableCapabilities,
+     * this metadata will list the supported high speed video size, fps range and max batch size
+     * configurations. All the sizes listed in this configuration will be a subset of the sizes
+     * reported by {@link android.hardware.camera2.params.StreamConfigurationMap#getOutputSizes }
+     * for processed non-stalling formats.</p>
+     * <p>For the high speed video use case, the application must
      * select the video size and fps range from this metadata to configure the recording and
      * preview streams and setup the recording requests. For example, if the application intends
      * to do high speed recording, it can select the maximum size reported by this metadata to
      * configure output streams. Once the size is selected, application can filter this metadata
      * by selected size and get the supported fps ranges, and use these fps ranges to setup the
      * recording requests. Note that for the use case of multiple output streams, application
-     * must select one unique size from this metadata to use. Otherwise a request error might
-     * occur.</p>
-     * <p>For normal video recording use case, where some application will NOT set
-     * {@link CaptureRequest#CONTROL_SCENE_MODE android.control.sceneMode} to HIGH_SPEED_VIDEO in capture requests, the fps ranges
-     * reported in this metadata must not be used to setup capture requests, or it will cause
-     * request error.</p>
+     * must select one unique size from this metadata to use (e.g., preview and recording streams
+     * must have the same size). Otherwise, the high speed capture session creation will fail.</p>
+     * <p>The min and max fps will be multiple times of 30fps.</p>
+     * <p>High speed video streaming extends significant performance pressue to camera hardware,
+     * to achieve efficient high speed streaming, the camera device may have to aggregate
+     * multiple frames together and send to camera device for processing where the request
+     * controls are same for all the frames in this batch. Max batch size indicates
+     * the max possible number of frames the camera device will group together for this high
+     * speed stream configuration. This max batch size will be used to generate a high speed
+     * recording request list by
+     * {@link android.hardware.camera2.CameraDevice#createConstrainedHighSpeedRequestList }.
+     * The max batch size for each configuration will satisfy below conditions:</p>
+     * <ul>
+     * <li>Each max batch size will be a divisor of its corresponding fps_max / 30. For example,
+     * if max_fps is 300, max batch size will only be 1, 2, 5, or 10.</li>
+     * <li>The camera device may choose smaller internal batch size for each configuration, but
+     * the actual batch size will be a divisor of max batch size. For example, if the max batch
+     * size is 8, the actual batch size used by camera device will only be 1, 2, 4, or 8.</li>
+     * <li>The max batch size in each configuration entry must be no larger than 32.</li>
+     * </ul>
+     * <p>The camera device doesn't have to support batch mode to achieve high speed video recording,
+     * in such case, batch_size_max will be reported as 1 in each configuration entry.</p>
+     * <p>This fps ranges in this configuration list can only be used to create requests
+     * that are submitted to a high speed camera capture session created by
+     * {@link android.hardware.camera2.CameraDevice#createConstrainedHighSpeedCaptureSession }.
+     * The fps ranges reported in this metadata must not be used to setup capture requests for
+     * normal capture session, or it will cause request error.</p>
      * <p><b>Range of valid values:</b><br></p>
-     * <p>For each configuration, the fps_max &gt;= 60fps.</p>
+     * <p>For each configuration, the fps_max &gt;= 120fps.</p>
      * <p><b>Optional</b> - This value may be {@code null} on some devices.</p>
      * <p><b>Limited capability</b> -
      * Present on all camera devices that report being at least {@link CameraCharacteristics#INFO_SUPPORTED_HARDWARE_LEVEL_LIMITED HARDWARE_LEVEL_LIMITED} devices in the
      * {@link CameraCharacteristics#INFO_SUPPORTED_HARDWARE_LEVEL android.info.supportedHardwareLevel} key</p>
      *
-     * @see CameraCharacteristics#CONTROL_AVAILABLE_SCENE_MODES
-     * @see CaptureRequest#CONTROL_SCENE_MODE
      * @see CameraCharacteristics#INFO_SUPPORTED_HARDWARE_LEVEL
      * @hide
      */
@@ -1308,7 +1327,8 @@ public final class CameraCharacteristics extends CameraMetadata<CameraCharacteri
      * the max pipeline depth.</p>
      * <p>A pipeline depth of X stages is equivalent to a pipeline latency of
      * X frame intervals.</p>
-     * <p>This value will be 8 or less.</p>
+     * <p>This value will normally be 8 or less, however, for high speed capture session,
+     * the max pipeline depth will be up to 8 x size of high speed capture request list.</p>
      * <p>This key is available on all devices.</p>
      *
      * @see CaptureResult#REQUEST_PIPELINE_DEPTH
@@ -1371,6 +1391,7 @@ public final class CameraCharacteristics extends CameraMetadata<CameraCharacteri
      *   <li>{@link #REQUEST_AVAILABLE_CAPABILITIES_BURST_CAPTURE BURST_CAPTURE}</li>
      *   <li>{@link #REQUEST_AVAILABLE_CAPABILITIES_YUV_REPROCESSING YUV_REPROCESSING}</li>
      *   <li>{@link #REQUEST_AVAILABLE_CAPABILITIES_DEPTH_OUTPUT DEPTH_OUTPUT}</li>
+     *   <li>{@link #REQUEST_AVAILABLE_CAPABILITIES_CONSTRAINED_HIGH_SPEED_VIDEO CONSTRAINED_HIGH_SPEED_VIDEO}</li>
      * </ul></p>
      * <p>This key is available on all devices.</p>
      *
@@ -1384,6 +1405,7 @@ public final class CameraCharacteristics extends CameraMetadata<CameraCharacteri
      * @see #REQUEST_AVAILABLE_CAPABILITIES_BURST_CAPTURE
      * @see #REQUEST_AVAILABLE_CAPABILITIES_YUV_REPROCESSING
      * @see #REQUEST_AVAILABLE_CAPABILITIES_DEPTH_OUTPUT
+     * @see #REQUEST_AVAILABLE_CAPABILITIES_CONSTRAINED_HIGH_SPEED_VIDEO
      */
     @PublicKey
     public static final Key<int[]> REQUEST_AVAILABLE_CAPABILITIES =
diff --git a/core/java/android/hardware/camera2/CameraMetadata.java b/core/java/android/hardware/camera2/CameraMetadata.java
index ac29f80..f8db6d9 100644
--- a/core/java/android/hardware/camera2/CameraMetadata.java
+++ b/core/java/android/hardware/camera2/CameraMetadata.java
@@ -648,6 +648,100 @@ public abstract class CameraMetadata<TKey> {
      */
     public static final int REQUEST_AVAILABLE_CAPABILITIES_DEPTH_OUTPUT = 8;
 
+    /**
+     * <p>The device supports constrained high speed video recording (frame rate &gt;=120fps)
+     * use case. The camera device will support high speed capture session created by
+     * {@link android.hardware.camera2.CameraDevice#createConstrainedHighSpeedCaptureSession }, which
+     * only accepts high speed request list created by
+     * {@link android.hardware.camera2.CameraDevice#createConstrainedHighSpeedRequestList }.</p>
+     * <p>A camera device can still support high speed video streaming by advertising the high speed
+     * FPS ranges in {@link CameraCharacteristics#CONTROL_AE_AVAILABLE_TARGET_FPS_RANGES android.control.aeAvailableTargetFpsRanges}. For this case, all normal
+     * capture request per frame control and synchronization requirements will apply to
+     * the high speed fps ranges, the same as all other fps ranges. This capability describes
+     * the capability of a specialized operating mode with many limitations (see below), which
+     * is only targeted at high speed video recording.</p>
+     * <p>The supported high speed video sizes and fps ranges are specified in
+     * {@link android.hardware.camera2.params.StreamConfigurationMap#getHighSpeedVideoFpsRanges }.
+     * To get desired output frame rates, the application is only allowed to select video size
+     * and FPS range combinations provided by
+     * {@link android.hardware.camera2.params.StreamConfigurationMap#getHighSpeedVideoSizes }.
+     * The fps range can be controlled via {@link CaptureRequest#CONTROL_AE_TARGET_FPS_RANGE android.control.aeTargetFpsRange}.</p>
+     * <p>In this capability, the camera device will override aeMode, awbMode, and afMode to
+     * ON, ON, and CONTINUOUS_VIDEO, respectively. All post-processing block mode
+     * controls will be overridden to be FAST. Therefore, no manual control of capture
+     * and post-processing parameters is possible. All other controls operate the
+     * same as when {@link CaptureRequest#CONTROL_MODE android.control.mode} == AUTO. This means that all other
+     * android.control.* fields continue to work, such as</p>
+     * <ul>
+     * <li>{@link CaptureRequest#CONTROL_AE_TARGET_FPS_RANGE android.control.aeTargetFpsRange}</li>
+     * <li>{@link CaptureRequest#CONTROL_AE_EXPOSURE_COMPENSATION android.control.aeExposureCompensation}</li>
+     * <li>{@link CaptureRequest#CONTROL_AE_LOCK android.control.aeLock}</li>
+     * <li>{@link CaptureRequest#CONTROL_AWB_LOCK android.control.awbLock}</li>
+     * <li>{@link CaptureRequest#CONTROL_EFFECT_MODE android.control.effectMode}</li>
+     * <li>{@link CaptureRequest#CONTROL_AE_REGIONS android.control.aeRegions}</li>
+     * <li>{@link CaptureRequest#CONTROL_AF_REGIONS android.control.afRegions}</li>
+     * <li>{@link CaptureRequest#CONTROL_AWB_REGIONS android.control.awbRegions}</li>
+     * <li>{@link CaptureRequest#CONTROL_AF_TRIGGER android.control.afTrigger}</li>
+     * <li>{@link CaptureRequest#CONTROL_AE_PRECAPTURE_TRIGGER android.control.aePrecaptureTrigger}</li>
+     * </ul>
+     * <p>Outside of android.control.*, the following controls will work:</p>
+     * <ul>
+     * <li>{@link CaptureRequest#FLASH_MODE android.flash.mode} (TORCH mode only, automatic flash for still capture will not
+     * work since aeMode is ON)</li>
+     * <li>{@link CaptureRequest#LENS_OPTICAL_STABILIZATION_MODE android.lens.opticalStabilizationMode} (if it is supported)</li>
+     * <li>{@link CaptureRequest#SCALER_CROP_REGION android.scaler.cropRegion}</li>
+     * <li>{@link CaptureRequest#STATISTICS_FACE_DETECT_MODE android.statistics.faceDetectMode} (if it is supported)</li>
+     * </ul>
+     * <p>For high speed recording use case, the actual maximum supported frame rate may
+     * be lower than what camera can output, depending on the destination Surfaces for
+     * the image data. For example, if the destination surface is from video encoder,
+     * the application need check if the video encoder is capable of supporting the
+     * high frame rate for a given video size, or it will end up with lower recording
+     * frame rate. If the destination surface is from preview window, the actual preview frame
+     * rate will be bounded by the screen refresh rate.</p>
+     * <p>The camera device will only support up to 2 high speed simultaneous output surfaces
+     * (preview and recording surfaces)
+     * in this mode. Above controls will be effective only if all of below conditions are true:</p>
+     * <ul>
+     * <li>The application creates a camera capture session with no more than 2 surfaces via
+     * {@link android.hardware.camera2.CameraDevice#createConstrainedHighSpeedCaptureSession }. The
+     * targeted surfaces must be preview surface (either from
+     * {@link android.view.SurfaceView } or {@link android.graphics.SurfaceTexture }) or
+     * recording surface(either from {@link android.media.MediaRecorder#getSurface } or
+     * {@link android.media.MediaCodec#createInputSurface }).</li>
+     * <li>The stream sizes are selected from the sizes reported by
+     * {@link android.hardware.camera2.params.StreamConfigurationMap#getHighSpeedVideoSizes }.</li>
+     * <li>The FPS ranges are selected from
+     * {@link android.hardware.camera2.params.StreamConfigurationMap#getHighSpeedVideoFpsRanges }.</li>
+     * </ul>
+     * <p>When above conditions are NOT satistied, the
+     * {@link android.hardware.camera2.CameraDevice#createConstrainedHighSpeedCaptureSession }
+     * and {@link android.hardware.camera2.CameraDevice#createConstrainedHighSpeedRequestList } will fail.</p>
+     * <p>Switching to a FPS range that has different maximum FPS may trigger some camera device
+     * reconfigurations, which may introduce extra latency. It is recommended that
+     * the application avoids unnecessary maximum target FPS changes as much as possible
+     * during high speed streaming.</p>
+     *
+     * @see CameraCharacteristics#CONTROL_AE_AVAILABLE_TARGET_FPS_RANGES
+     * @see CaptureRequest#CONTROL_AE_EXPOSURE_COMPENSATION
+     * @see CaptureRequest#CONTROL_AE_LOCK
+     * @see CaptureRequest#CONTROL_AE_PRECAPTURE_TRIGGER
+     * @see CaptureRequest#CONTROL_AE_REGIONS
+     * @see CaptureRequest#CONTROL_AE_TARGET_FPS_RANGE
+     * @see CaptureRequest#CONTROL_AF_REGIONS
+     * @see CaptureRequest#CONTROL_AF_TRIGGER
+     * @see CaptureRequest#CONTROL_AWB_LOCK
+     * @see CaptureRequest#CONTROL_AWB_REGIONS
+     * @see CaptureRequest#CONTROL_EFFECT_MODE
+     * @see CaptureRequest#CONTROL_MODE
+     * @see CaptureRequest#FLASH_MODE
+     * @see CaptureRequest#LENS_OPTICAL_STABILIZATION_MODE
+     * @see CaptureRequest#SCALER_CROP_REGION
+     * @see CaptureRequest#STATISTICS_FACE_DETECT_MODE
+     * @see CameraCharacteristics#REQUEST_AVAILABLE_CAPABILITIES
+     */
+    public static final int REQUEST_AVAILABLE_CAPABILITIES_CONSTRAINED_HIGH_SPEED_VIDEO = 9;
+
     //
     // Enumeration values for CameraCharacteristics#SCALER_CROPPING_TYPE
     //
@@ -1725,6 +1819,10 @@ public abstract class CameraMetadata<TKey> {
     public static final int CONTROL_SCENE_MODE_BARCODE = 16;
 
     /**
+     * <p>This is deprecated, please use
+     * {@link android.hardware.camera2.CameraDevice#createConstrainedHighSpeedCaptureSession }
+     * and {@link android.hardware.camera2.CameraDevice#createConstrainedHighSpeedRequestList }
+     * for high speed video recording.</p>
      * <p>Optimized for high speed video recording (frame rate &gt;=60fps) use case.</p>
      * <p>The supported high speed video sizes and fps ranges are specified in
      * android.control.availableHighSpeedVideoConfigurations. To get desired
@@ -1799,6 +1897,7 @@ public abstract class CameraMetadata<TKey> {
      * @see CaptureRequest#SCALER_CROP_REGION
      * @see CaptureRequest#STATISTICS_FACE_DETECT_MODE
      * @see CaptureRequest#CONTROL_SCENE_MODE
+     * @deprecated Please refer to this API documentation to find the alternatives
      */
     public static final int CONTROL_SCENE_MODE_HIGH_SPEED_VIDEO = 17;