Android Camera Subsystem — basic image processing steps done at the hardware level in Android Camera
Towards the end of the last decade camera has emerged as one of the key factors that contribute towards smartphone sales and different manufacturers are trying to stay at the top of the throne. While everyone loves using a camera — the set of things that happens in a different layer of the hardware and software stack of Android Camera is not very popular knowledge. The design, implementation, and usage of these systems govern metrics like image quality and camera performance (latency, throughput, etc).
In this post, I have mentioned various algorithms implemented in the android camera subsystem that is consistent across OEMs which are used to produce stunning images that we capture from camera applications. I have covered each processing step in much more details in my blog article — Basic image processing steps are done at the hardware level in Android Camera (blog.minhazav.dev)
Legends
- HAL: Hardware Access Layer
- ISP: Image Signal Processor
- DSP: Digital Signal Processor
- 3A: Auto Exposure, Auto Focus, Auto White-Balance
- OEM: Original Equipment Manufacturers
- JPEG: Compressed image format — Wikipedia
- YUV: An uncompressed image format — Wikipedia
Android Camera Stack
Camera Subsystem in Android
As per source.android.com on Android Camera:
Android’s camera hardware abstraction layer (HAL) connects the higher-level camera framework APIs in Camera 2 to your underlying camera driver and hardware. The camera subsystem includes implementations for camera pipeline components while the camera HAL provides interfaces for use in implementing your version of these components.
The camera subsystem includes the implementation of fundamental image processing algorithms that converts RAW Bayer output from the camera sensor to a full-fledged image that can be consumed by applications and users. In another article, I have attempted to explain — Android camera hardware architecture.
Camera Sensor and RAW output
As described in this post the output of the camera sensor only contains information on one color per pixel as compared to three-color (RGB) per pixel we intuitively assume.
Figure: Output from the camera sensor.
The output from camera sensors is preserved in a format called RAW. The RAW image is often called digital negatives. A RAW image can only be consumed by specialized image viewing tools but is popular amongst photographers as it preserves the image as it was captured by Camera without losing information.
The image output from HAL is consumed by applications and most OEMs add support for JPEG and YUV image as output from the camera, while some manufacturers add support for returning RAW image directly as well. You can think of YUV as a processed image that is ready to be consumed. While JPEG is a compressed image format that represents a compressed version of the processed image.
Image Processing done in Camera Subsystem
- Hot pixel correction
- Demosaic
- Noise reduction
- Shading correction
- Geometric correction
- Color correction
- Tone Curve adjustment
- Edge enhancement
Interesting Notes
All of these steps are not mandatory steps between image capture and image available at the application layer. The set of algorithms applied in the pipeline has a trade-off with frame rate and can be configured using CaptureRequest parameters. There are three modes:
OFF: This processing block is disabled. The demosaic, color correction and tone curve adjustment blocks cannot be disabled.FAST: In this mode, the processing block may not slow down the output frame rate compared to OFF mode, but should otherwise produce the best-quality output it can given that restriction. Typically, this would be used for preview or video recording modes, or burst capture for still images. The set of algorithms applied depends on the processing capability of the device.HIGH_QUALITY: In this mode, the processing block should produce the best quality result possible, slowing down the output frame rate as needed.
