Bringing Machine Learning in Android with MediaPipe - DroidJam 2023

Transcript

1. @arif_faizin
   Curriculum Developer Lead,
   Dicoding Indonesia
   Bringing Machine Learning
   in Android with MediaPipe
   DroidJam 2023
   

   View Slide

2. Let’s Start with Why~
   DroidJam 2023
   

   View Slide

3. View Slide

4. View Slide

5. DroidJam 2023
   65% of consumers have trust
   in the businesses which use
   AI technology
   - Forbes Advisor
   

   View Slide

6. View Slide

7. Intro to ML
   DroidJam 2023
   

   View Slide

8. Kai
   Anak MD Bangkit yang ibunya punya Toko Kelontong
   

   View Slide

9. View Slide

10. Lengkuas Jahe
    Kencur Kunyit
    

    View Slide

11. Traditional
    Programming
    Rules
    Input Output
    

    View Slide

12. Machine
    Learning
    Output
    Input Rules
    

    View Slide

13. View Slide

14. How to?
    DroidJam 2023
    

    View Slide

15. On-Cloud On-Device
    
    

    View Slide

16. ● Lower latency & close knit interactions
    ● Offline availability
    ● Privacy preserving
    ● Cost savings
    On-Device
    
    

    View Slide

17. On-Device ML
    From Scratch 🤔
    

    View Slide

18. 
    On-Device ML
    Using Framework 😎
    

    View Slide

19. Alternative Framework
    On-Device ML
    

    View Slide

20. View Slide

21. View Slide

22. View Slide

23. Deep Dive
    into MediaPipe
    DroidJam 2023
    

    View Slide

24. Model is at the core of
    an on-device ML solution
    Model Inference
    Output
    Input
    ML app
    ● Customized to the ad-hoc use cases
    ● Light-weight and efficient
    ● Target to hardware
    ● Sparsified for best performance
    

    View Slide

25. ML Pipeline streamlines
    the process from raw
    inputs to output results Model Inference
    Output
    Live Camera
    ML app
    Flow Control
    Post-processing
    Synchronization
    Data Preprocessing
    ● Domain-specific processing (e.g. vision / NLP / audio)
    ● E2E acceleration across CPU / GPU / EdgeTPU / DSP
    ● Cross-platform deployment to Android / iOS, web,
    baremetal
    

    View Slide

26. Display
    Live Camera
    ML app
    Buffer Management
    Format Conversion
    Image Filtering
    Data Subsampling
    Data Lifecycle
    Timestamp Extraction
    Timestamp Alignment
    Thread Management GPU/CPU Data Transfer
    Multi-threaded GPU Compute
    iOS Metal
    OpenGL ES
    Trace Collection
    Performance Profiling
    C++ Programming
    Resource Caching
    Asset Loading
    GPU Timing Measurement
    Cross-platform Abstraction
    Data Marshalling
    CPU Affinity
    Java Native Interface
    Model Inference
    Flow Control
    Post-processing
    Synchronization
    Data Preprocessing
    Both involves a lot of
    complexity that
    hinders
    fast development
    

    View Slide

27. MediaPipe abstracts
    this complexity into
    MediaPipe Tasks
    Model Inference
    Display
    MediaPipe Tasks
    Live Camera
    ML app
    Flow Control
    Post-processing
    Synchronization
    Data Preprocessing
    

    View Slide

28. … while meeting your
    custom modeling
    needs with
    MediaPipe Model Maker
    Model Inference
    Display
    MediaPipe
    Model Maker
    Live Camera
    ML app
    Flow Control
    Post-processing
    Synchronization
    Data Preprocessing
    Custom model
    

    View Slide

29. No-code GUI with MediaPipe Studio
    

    View Slide

30. View Slide

31. DroidJam 2023
    

    View Slide

32. Create Image
    Classification App
    DroidJam 2023
    

    View Slide

33. DroidJam 2023
    ● Create App to get image from Gallery or Camera
    ● Alternative solution
    ○ Gallery
    ■ PhotoPicker
    ActivityResultContracts.PickVisualMedia()
    ■ Intent ACTION_GET_CONTENT
    ■ Intent ACTION_PICK
    ○ Camera
    ■ Intent ACTION_IMAGE_CAPTURE
    ■ ActivityResultContracts.TakePicture()
    ■ CameraX
    Starter Project
    

    View Slide

34. View Slide

35. View Slide

36. View Slide

37. View Slide

38. DroidJam 2023
    // 0. build.gradle.kts
    implementation("com.google.mediapipe:tasks-vision:xxx")
    

    View Slide

39. DroidJam 2023
    // 1. Setup Image Classifier (ImageClassifierHelper.kt)
    val baseOptionsBuilder = BaseOptions.builder()
    .setDelegate(Delegate.GPU) // CPU, GPU
    .setModelAssetPath(MODEL_PATH)
    val optionsBuilder = ImageClassifier.ImageClassifierOptions.builder()
    .setScoreThreshold(0.1f) // minimum 10%
    .setMaxResults(3)
    .setRunningMode(RunningMode.IMAGE)
    .setBaseOptions(baseOptionsBuilder.build())
    val options = optionsBuilder.build()
    val imageClassifier = ImageClassifier.createFromOptions(context, options)
    

    View Slide

40. DroidJam 2023
    // 2. Create instance of ImageClassifierHelper
    // Get Data from Camera (in Activity)
    // Convert Uri to Bitmap
    imageUri?.let { uri ->
    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.P) {
    val source = ImageDecoder.createSource(contentResolver, imageUri)
    ImageDecoder.decodeBitmap(source)
    } else {
    MediaStore.Images.Media.getBitmap(contentResolver, uri)
    }.copy(Bitmap.Config.ARGB_8888, true)?.let { bitmap ->
    imageClassifierHelper.classifyImage(bitmap)
    }
    }
    

    View Slide

41. Source: https://twitter.com/equasys_de/status/754975190459834368/photo/1
    

    View Slide

42. DroidJam 2023
    // 3. Convert the input Bitmap object to an MPImage object to run inference
    // ImageClassifierHelper.kt
    fun classifyImage(bitmap: Bitmap) {
    val mpImage: MPImage = BitmapImageBuilder(bitmap).build()
    val imageProcessingOptions = ImageProcessingOptions.builder().build()
    val startTime = SystemClock.uptimeMillis()
    imageClassifier?.classify(mpImage, imageProcessingOptions).also { result ->
    val inferenceTime = SystemClock.uptimeMillis() - startTime
    imageClassifierListener?.onResults(result, inferenceTime)
    }
    if (imageClassifier == null) {
    imageClassifierListener?.onError(
    "Image classifier failed to classify."
    )
    }
    }
    

    View Slide

43. // Output
    [Classifications
    {categories=
    [
    (displayName=
    score=0.41453125
    index=621)>,
    (displayName=
    score=0.35921875
    index=509)>
    ],
    headIndex=0,
    headName=Optional[probability]
    }
    ]
    Note:
    It’s only works in real devices, not in Emulator
    

    View Slide

44. New skill unlocked!
    

    View Slide

45. Real Time
    Classification?
    DroidJam 2023
    

    View Slide

46. DroidJam 2023
    val cameraProviderFuture = ProcessCameraProvider.getInstance(this)
    cameraProviderFuture.addListener({
    val cameraProvider = cameraProviderFuture.get()
    val preview = Preview.Builder()
    .setTargetAspectRatio(AspectRatio.RATIO_4_3)
    .build()
    .also {
    it.setSurfaceProvider(binding.viewFinder.surfaceProvider)
    }
    cameraProvider.unbindAll()
    cameraProvider.bindToLifecycle(
    this,
    CameraSelector.DEFAULT_BACK_CAMERA,
    preview
    )
    }, ContextCompat.getMainExecutor(this))
    ● Request Permission CAMERA
    ● CameraX implementation
    Starter Project
    

    View Slide

47. DroidJam 2023
    // 1. Setup Options Configuration (ImageClassifierHelper.kt)
    val optionsBuilder = ImageClassifier.ImageClassifierOptions.builder()
    .setScoreThreshold(0.1f)
    .setMaxResults(3)
    .setRunningMode(RunningMode.LIVE_STREAM) // IMAGE, VIDEO, LIVE_STREAM
    .setBaseOptions(baseOptionsBuilder.build())
    if (runningMode == RunningMode.LIVE_STREAM) {
    optionsBuilder.setResultListener(this::returnLivestreamResult)
    optionsBuilder.setErrorListener(this::returnLivestreamError)
    }
    

    View Slide

48. DroidJam 2023
    // 2. Setup ImageAnalysis for CameraX (in Activity)
    val imageAnalyzer = ImageAnalysis.Builder()
    .setTargetAspectRatio(AspectRatio.RATIO_4_3)
    .setTargetRotation(binding.viewFinder.display.rotation)
    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
    .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888)
    .build()
    .also {
    it.setAnalyzer(Executors.newSingleThreadExecutor()) { image ->
    imageClassifierHelper.classifyLiveStreamFrame(image)
    }
    }
    ...
    cameraProvider.bindToLifecycle(
    this,
    CameraSelector.DEFAULT_BACK_CAMERA,
    preview,
    imageAnalyzer
    )
    

    View Slide

49. DroidJam 2023
    // 3. Convert the input Bitmap object to an MPImage object to run inference
    fun classifyLiveStreamFrame(image: ImageProxy) {
    ...
    val mpImage = BitmapImageBuilder(bitmapBuffer).build()
    // Used for rotating the frame image so it matches our models
    val imageProcessingOptions = ImageProcessingOptions.builder()
    .setRotationDegrees(image.imageInfo.rotationDegrees)
    .build()
    val frameTime = SystemClock.uptimeMillis()
    // Run inference
    imageClassifier?.classifyAsync(mpImage, imageProcessingOptions, frameTime)
    }
    

    View Slide

50. “Your focus determines
    your reality.”
    – Qui-Gon Jinn
    

    View Slide

51. Object Detection:
    A new Hope
    DroidJam 2023
    

    View Slide

52. View Slide

53. View Slide

54. DroidJam 2023
    // 1. Setup Object Detector (ObjectDetectorHelper.kt)
    val baseOptionsBuilder = BaseOptions.builder()
    .setDelegate(Delegate.GPU) // CPU, GPU
    .setModelAssetPath(MODEL_PATH)
    val optionsBuilder = ObjectDetector.ObjectDetectorOptions.builder()
    .setScoreThreshold(0.1f)
    .setMaxResults(3)
    .setRunningMode(RunningMode.LIVE_STREAM)
    .setBaseOptions(baseOptionsBuilder.build())
    val options = optionsBuilder.build()
    val imageClassifier = ImageClassifier.createFromOptions(context, options)
    

    View Slide

55. DroidJam 2023
    // 2. Setup ImageAnalysis for CameraX (in Activity)
    val imageAnalyzer = ImageAnalysis.Builder()
    .setTargetAspectRatio(AspectRatio.RATIO_16_9) // adjust with model
    .setTargetRotation(binding.viewFinder.display.rotation)
    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
    .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888)
    .build()
    .also {
    it.setAnalyzer(Executors.newSingleThreadExecutor()) { image ->
    objectDetectorHelper.detectLivestreamFrame(image)
    }
    }
    

    View Slide

56. DroidJam 2023
    // 3. Convert the input Bitmap object to an MPImage object to run inference
    fun detectLiveStreamFrame(image: ImageProxy) {
    ...
    val mpImage = BitmapImageBuilder(bitmapBuffer).build()
    // Used for rotating the frame image so it matches our models
    val imageProcessingOptions = ImageProcessingOptions.builder()
    .setRotationDegrees(image.imageInfo.rotationDegrees)
    .build()
    val frameTime = SystemClock.uptimeMillis()
    // Run inference
    objectDetector?.detectAsync(mpImage, imageProcessingOptions, frameTime)
    }
    

    View Slide

57. New skill unlocked!
    

    View Slide

58. DroidJam 2023
    // 4a. Draw Box Using Custom View in XML
    class OverlayView(context: Context?, attrs: AttributeSet?) :
    View(context, attrs) {
    override fun draw(canvas: Canvas) {
    super.draw(canvas)
    // Draw bounding box around detected objects
    val drawableRect = RectF(left, top, right, bottom)
    canvas.drawRect(drawableRect, boxPaint)
    // Draw text for detected object
    canvas.drawText(
    drawableText,
    left,
    top + bounds.height(),
    textPaint
    )
    }
    }
    }
    

    View Slide

59. DroidJam 2023
    // 4b. Create Box Using Composable
    @Composable
    fun ResultsOverlay(...) {
    val detections = results.detections()
    if (detections != null) {
    for (detection in detections) {
    ...
    Box(
    modifier = Modifier
    .border(3.dp, Turquoise)
    .width(boxWidth.dp)
    .height(boxHeight.dp)
    )
    Box(modifier = Modifier.padding(3.dp)) {
    Text(
    text = resultText,
    modifier = Modifier
    .background(Color.Black)
    .padding(5.dp, 0.dp),
    color = Color.White,
    )
    }
    }
    }
    

    View Slide

60. Time Skip~
    DroidJam 2023
    

    View Slide

61. 95% Kunyit
    ML
    CC
    CC
    MD
    Rp49.999
    86% Kencur
    Rp14.045
    

    View Slide

62. Summary
    DroidJam 2023
    

    View Slide

63. Emang boleh bikin aplikasi AI
    segampang ini?
    DroidJam 2023
    

    View Slide

64. Try the others~
    

    View Slide

65. DroidJam 2023
    References
    ● MediaPipe Documentation
    ● Introducing MediaPipe for On-Device Machine Learning
    ● Introduction to ML on Android with MediaPipe
    ● Easy on-device Machine Learning with MediaPipe
    ● ML Kit: Turnkey APIs to use on-device ML in mobile apps | Session
    ● What's new in Machine Learning for Google Developers
    

    View Slide

66. "Does my dream have to be success?
    Can’t it be a person?”
    – Nam Do San
    

    View Slide

67. DroidJam 2023
    Hatur nuwun!
    Deck is available at
    https://speakerdeck.com/arifaizin
    

    View Slide