1. Cosc 5/4730 Mobile Media API (MMAPI) v1.1 JSR 135 and Android media Part 2: Pictures and Video

2. Emulators The emulators all have problems, much of the following code has been tested on the actual phones Video playback code works, but the video may not always display in the emulators android – Video/picture capture does not work as documented. Blackberry – Camera, you can choose a picture with the emulator to use or some emulators will use your background image – Video capture does not work as documented.

3. PLAY MEDIA Blackberry MMAPI

4. Video formats. MP4 – H.264, MPEG-4 with simple profile level 3, and H.263 AVI (MPEG4 profile level 3) WMV (simple profile, WMV3) at most the video resolution can be the size screen of the device. – 30 frames per second

5. Playing video Similar to audio – use the Player, realize, play, etc… – But you need a “View Player field” same idea of the “View finder” when we get to taking a picture – We get the viewer from the VideoControl object. – We can use the MediaKeyListener for “media” key events (assuming the phone has them), but there is no easy way to display controls, like play, pause, etc..

6. The Basics Just like audio, get the player and realize() player = javax.microedition.media.Manager.createPlayer( "file:///SDCard/BlackBerry/videos/the-empire.3gp"); player.realize(); Get the VideoControl pieces and get a “View Player field” VideoControl videoControl = (VideoControl) player.getControl( "VideoControl"); Field videoField = (Field) videoControl.initDisplayMode( VideoControl.USE_GUI_PRIMITIVE, "net.rim.device.api.ui.Field" ); add the field to the screen add(videoField); I found it didn’t work unless I went fullscreen. audio worked, but no video videoControl.setDisplayFullScreen(true); start playing. player.start();

7. MediaKeyListener PlayListener Use the PlayListener so we figure out when the media has finished and be able to close down stuff. – Plus deal with other issues as necessary use the MediaKeyListener to deal with “standard” blackberry media buttons as well.

8. For simplicity… Since I had to run in full screen – I added menus to play, pause, stop. – volume up/down are handled by the MediaKeysListener, plus the rest media keys too. Except they don’t exist on the phones I have access too. The mute button on many BB is a play/pause toggle button. Including bold and storm.

9. Volume Control Handled via a VolumeControl object. volume = (VolumeControl) player.getControl("VolumeControl"); int getlevel() and setLevel(int) setMute(boolean) – true mute, false unmute;

10. seeking player.getMediaTime() returns a long that represents the spot in the media – 1,000,000 is about 1 sec For a 102 second video, the duration is 102,898,000 – since I can’t see smaller then a second, assuming the video is 102 seconds + almost 1 more second. player.setMediaTime(long) allows you to change the position it is playing.

11. Buffering. Get content off the web – like I do with many examples… except buffering doesn’t work. – The advanced Blackberry Development book has an example of how to create a streaming video buffer from none streaming media, such mp4 and 3gp files.

12. Code example To run video player example – Note, it assumes you have an SDCARD and that file:///SDCard/BlackBerry/videos/the-empire.3gp this file exists. – It’s in the /res/ directory.

13. Using native player Blackberry has a native media player. You can launch it to play audio/video – Remember you lose control and you app is now in the background. First get the registry with you full class name. Registry reg = Registry.getRegistry( "edu.cs4755.VideoCapture"); – Then create a invocation of the file you want to play and invoke it. Invocation invocation = new Invocation( "file:///SDCard/Blackberry/videos/testrecording.3gp"); reg.invoke(invocation); If this all reminds you of a Android, it’s very close an intent.

14. PLAY MEDIA Android android.media

15. Supported formats In general Android’s support is consistent with other mobile phones. It supports the 3GP (.3gp) and MPEG-4 (.mp4) file formats. – 3GP is a video standard derived from MPEG-4 specifically for use by mobile devices. As far as codecs go, Android supports H.263, a codec designed for low-latency and low-bitrate videoconferencing applications. H.263 video is supported in either MPEG-4 (.mp4) or 3GP (.3gp) files. Android also supports MPEG-4 Simple Profile in 3GP files (.3gp) as well as H.264.

16. Android First method – For greater flexibility you will need to use the mediaPlayer and a surfaceView MediaPlayer like the audio and use a surfaceView to display the video. There are examples in the API demo, plus several of the books. The second method uses a VideoView to display. – The VideoView uses a MediaController to start and stop and provide functions to control the video play back. – With a MediaPlayer like the audio. prepare(), then start() – This is one I’ll cover in this lecture.

17. VideoView VideoView is a View that has video playback capabilities and can be used directly in a layout. We can then add controls (play, pause, forward, back, etc) with the MediaController.

18. ViewView example Get the ViewView out of the layout vv = (VideoView) this.findViewById( R.id.VideoView); Setup where the file to play is Uri videoUri = Uri.parse(Environment.getExternalStorageDirectory ().getPath() + "/example.mp4"); vv.setVideoURI(videoUri); play the video vv.start();

19. Adding media controllers. Very simple vv = (VideoView) this.findViewById(R.id.VideoView); vv.setMediaController(new MediaController(this)); Now media controls will show up on the screen.

20. Using native media player Call for an Intent and send it. Uri data = Uri.parse(VideoFile); intent.setDataAndType(data, "video/mp4"); startActivity(intent); Remember, your app is now in the background.

21. DISPLAYING A PICTURE Blackberry and Android

22. Displaying Pictures For both android and blackberry – See code already covered to how display pictures. bitmapfield, ImageView, etc…

23. TAKING A PICTURE Blackberry MMAPI

24. Taking a picture. Taking a picture requires that you get a player and then VideoControl, finally create what amounts a view finder. – Then you can take the picture, which comes back as bytes (but in the format you spec’d, such as jpeg) – After that you do what you want with it.

25. Example First get the player part. try { Player player = Manager.createPlayer( "capture://video?encoding=jpeg" ); player.realize(); player.start(); Now get the Video Control VideoControl videoControl = (VideoControl)player.getControl( "javax.microedition.media.control.VideoControl"); Create a “View Finder” Field CameraView = (Field)videoControl.initDisplayMode( VideoControl.USE_GUI_PRIMITIVE, "net.rim.device.api.ui.Field"); add(CameraView); //display the viewfinder, but you don’t have too. } catch (Exception e) { System.out.println("Error"+e.getMessage()); e.printStackTrace(); }

26. Capture:// My code used the default, but you can specify "encoding=jpeg&width=1600&height=1200&quality=superfine" "encoding=jpeg&width=1600&height=1200&quality=fine" "encoding=jpeg&width=1600&height=1200&quality=normal" "encoding=jpeg&width=1024&height=768&quality=superfine" "encoding=jpeg&width=1024&height=768&quality=fine" "encoding=jpeg&width=1024&height=768&quality=normal" "encoding=jpeg&width=640&height=480&quality=superfine" "encoding=jpeg&width=640&height=480&quality=fine" "encoding=jpeg&width=640&height=480&quality=normal" The supported imageType(s) can also be queried by invoking System.getProperty("video.snapshot.encodings")

27. “View Finder” This line requires a bit of explaining – Field CameraView = (Field) videoControl.initDisplayMode( VideoControl.USE_GUI_PRIMITIVE, "net.rim.device.api.ui.Field"); InitDisplayMode has two options: – USE_GUI_PRIMITIVE In this mode, arg can be the fully qualified class name of primitive GUI components, for example javax.microedition.lcdui.Item or net.rim.device.api.ui.Field. Depending on the value of arg, initDisplay returns an Item or a Field. – USE_DIRECT_VIDEO When this mode is used, arg must be the fully qualified class name of Canvas, for example javax.microedition.lcdui.Canvas or a subclass of it. In this case, initDisplay will return an instance of Canvas and the video will be directly rendered on this Canvas. Then we cast it as field to display it as specified in the arguments.

28. Get the picture Finally we can now take the picture. – Use a method called Snapshot – byte picData[] = videoControl.getSnapshot(null); Null says take the default (or whatever we already specified) and use the default camera settings. You could again, specify the image quality in the getSnapshot parameters. To get the image into a bitmap object – image = Bitmap.createBitmapFromBytes(picData, 0, picData.length, 1);

29. TAKING A PICTURE Android

30. What to use Android packages import android.hardware.CameraDevice; import android.hardware.CameraDevice.CaptureParams; – Permissions and features This too, if you change the how the camera is functioning.

31. Taking a picture In brief – CameraDevice cameraDevice = CameraDevice.open() – To preview you need a surfaceHolder then use setPreviewDisplay(surfaceHolder) and cameraDevice.startPreview() – Finally use the takePicture(…) to get the picture – release() and close() the CameraDevice to release it.

32. “View Finder” The view finder is implemented via a SurfaceHolder and SurfaceView – In the layout, the a surfaceView is used. Example: public class PicCapture extends Activity implements OnClickListener, SurfaceHolder.Callback, Camera.PictureCallback { … cameraView = (SurfaceView) this.findViewById(R.id.CameraView); surfaceHolder = cameraView.getHolder(); surfaceHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS); surfaceHolder.addCallback(this); To finally take the picture we need all this too. cameraView.setFocusable(true); cameraView.setFocusableInTouchMode(true); cameraView.setClickable(true); cameraView.setOnClickListener(this);

33. “View Finder” (2) The code to implement the surfaceHolder can be very simple. When created open the camera and set the display surfaceCreated() { camera = Camera.open(); try { camera.setPreviewDisplay(holder); catch (IOException exception) { camera.release(); } } Once it’s ready, start the preview surfaceChanged() { camera.startPreview(); }

34. “View Finder” (3) When we are done surfaceDestroyed() { camera.stopPreview(); camera.release(); }

35. Get the Picture Using the Camera.PictureCallBack we implement, we can get the data for the picture and decide what to do it with it. In its simplest form, which doesn’t nothing with the picture public void onPictureTaken(byte[] data, Camera camera) { byte[] data is the picture that was taken this just restarts the preview. camera.startPreview(); }

36. Get the Picture (2) To take the picture we use the camera.takePicture method in the onClick method for the SurfaceView public void onClick(View v) { camera.takePicture(null, null, null, this); } takePicture (Camera.ShutterCallback shutter, Camera.PictureCallback raw, Camera.PictureCallback postview, Camera.PictureCallback jpeg) We only need the last to get the picture and it show on the previous slide. – shutter the callback for image capture moment, or null – raw the callback for raw (uncompressed) image data, or null – postview the callback with postview image data, may be null – jpeg the callback for JPEG image data, or null

37. Android Example Code A note for the example code – The example code will put any pictures taken into the “Camera roll” and can be seen with the gallery app. – Remember it uses touch to take the picture.

38. RECORDING VIDEO Android android.media

39. First… Most examples and code on the web and from the android books, DO NOT work. – lots of subtle errors debugging is made more difficult, because – the camera throws errors CameraInput Recording is not ready … frame dropped. – the AudioFlinger shows constant buffer overflow warnings. – And this is when the app is working correctly.

40. Uses a surfaceView Like a taking a picture, we need a view finder which uses a surfaceView. Call for the MediaRecorder and setup the encoding. – both audio and video.

41. Example SurfaceView in onCreate recorder = new MediaRecorder(); //setup recorder settings Next Slide SurfaceView cameraView = (SurfaceView) findViewById(R.id.CameraView); holder = cameraView.getHolder(); holder.addCallback(this); holder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS); implemented methods. public void surfaceCreated(SurfaceHolder holder) { recorder.setPreviewDisplay(holder.getSurface()); recorder.prepare(); }

42. Example SurfaceView (2) public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) { } public void surfaceDestroyed(SurfaceHolder holder) { if (recording) { recorder.stop(); recording = false; } recorder.release(); finish(); }

43. Recorder Settings We need to set the sources for audio and video recorder.setAudioSource( MediaRecorder.AudioSource.MIC); recorder.setVideoSource( MediaRecorder.VideoSource.DEFAULT); – MediaRecorder.VideoSource.CAMERA should work as well. Now we need to setup encoders. – In android 2.2 we can use profiles instead of setting everything manually. – CamcorderProfile.QUALITY_HIGH or QUALITY_LOW CamcorderProfile cpHigh = CamcorderProfile.get(CamcorderProfile.QUALITY_HIGH); recorder.setProfile(cpHigh); And set the output location recorder.setOutputFile("/sdcard/videocapture_example.mp4");

44. Recorder Settings (2) Manual settings could look like this: recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4); recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB); recorder.setVideoEncoder(MediaRecorder.VideoEncoder.MPEG_4_SP); QUALITY_HIGH settings are MP4 file QUALITY_LOW settings are 3gp file The manual list is very long, see the android doc’s or Apress - Pro Android Media Developing Graphics, Music, Video, and Rich Media Apps for Smartphones and Tablets, Chapter 11 for a full list of settings.

45. Recording To record recorder.start(); To stop recorder.stop(); – The file should be there at this point. Remember when you are done recorder.release();

46. Example code The example code – need an Sdcard to test code and the file will be located /sdcard/videocapture_example.mp4 The code uses an extended surfaceView call captureSurface, instead of just a surfaceView – The code is all there, but rearranged from the slides. Honesty, the code just didn’t work without an extended SurfaceView The code starts up with the viewfinder, touch the screen to start recording, again to stop recording. It will then launch the native media player to replay the video.

47. RECORDING VIDEO Blackberry MMAPI

48. Video Recording. Use all the same things from pic capturing – need a “view finder” – We the capture line And RecordControl – use startRecord() to start it (needs be a thread) – use stopRecord() to pause recording – use commit() to finish it. Player – choose to setRecordLocation or stream.

49. Example code To get the “view finder” going player = Manager.createPlayer( "capture://video?encoding=video/3gpp&mode=standard" ); player.realize(); player.start(); videoControl = (VideoControl)player.getControl( "VideoControl"); recorder = (RecordControl)player.getControl("RecordControl"); CameraView = (Field)videoControl.initDisplayMode( VideoControl.USE_GUI_PRIMITIVE, "net.rim.device.api.ui.Field"); add(CameraView);

50. Capture line capture://video?encoding=video/3gpp&mode=standar d – standard video based on specifications of the phone capture://video?encoding=video/3gpp&mode=mms – creates a video based on MMS standards intended for sending the video via mms You can ask the phone which encoding it can use with: – String encodingsString = System.getProperty( "video.encodings" ); – And parse the string.

51. Encodings from the Bold 9000 encoding=video/3gpp&mode=standard encoding=video/3gpp&mode=mms encoding=video/3gpp&width=480&height=320&video_codec=MPEG-4&audio_codec=AMR encoding=video/3gpp&width=176&height=144&video_codec=MPEG-4&audio_codec=AMR encoding=video/3gpp&width=480&height=320&video_codec=MPEG-4&audio_codec=PCM encoding=video/3gpp&width=176&height=144&video_codec=MPEG-4&audio_codec=PCM encoding=video/3gpp&width=480&height=320&video_codec=H263&audio_codec=AMR encoding=video/3gpp&width=176&height=144&video_codec=H263&audio_codec=AMR encoding=video/3gpp&width=480&height=320&video_codec=H263&audio_codec=PCM encoding=video/3gpp&width=176&height=144&video_codec=H263&audio_codec=PCM encoding=video/3gpp&width=480&height=320&video_codec=H264&audio_codec=AMR encoding=video/3gpp&width=176&height=144&video_codec=H264&audio_codec=AMR encoding=video/3gpp&width=480&height=320&video_codec=H264&audio_codec=PCM encoding=video/3gpp&width=176&height=144&video_codec=H264&audio_codec=PCM

52. Start Recording. Once the view finder is up and running Note the view finder fails in a popup, I had to use a MainScreen to get it to work. Set the location recorder.setRecordLocation("file:///SDCard/Blac kberry/videos/testrecording.3gp"); and start recording. recorder.startRecord();

53. finish recording. stopRecord() – pauses the recording. You can run startRecord() again to continue recording commit() – finishes and write outs the find file (assuming location used) Then you can close the player.

54. Recording to a stream. Use the connector.open to provide a stream get the outputsrearm and use setRecordStream fc = (FileConnection)Connector.open( STREAM_VIDEO_FILE ); rc.setRecordStream(fc.openOutputStream()); After you issue commit() close the outputstream

55. Example code. Two examples provided – VideoCapture.zip very basic app to record video Launches the native player to play the video after recording. – VideoRecordingApplication.zip more complex (writing by blackberry) – Note the videoplay code doesn’t work. Both require an Sdcard, since they store the video on the card.

56. References Blackberry – http://docs.blackberry.com/en/developers/deliverables/11942/Create_BB_ap p_disp_image_from_camera_viewFinder_739696_11.jsp http://docs.blackberry.com/en/developers/deliverables/11942/Create_BB_ap p_disp_image_from_camera_viewFinder_739696_11.jsp – http://docs.blackberry.com/en/developers/deliverables/17968/Take_a_pictur e_in_a_BB_device_app_1228201_11.jsp http://docs.blackberry.com/en/developers/deliverables/17968/Take_a_pictur e_in_a_BB_device_app_1228201_11.jsp – http://www.leterry.com/how-to-take-a-snapshot-using-the-built-in-camera-of- a-blackberry-smartphone.html http://www.leterry.com/how-to-take-a-snapshot-using-the-built-in-camera-of- a-blackberry-smartphone.html – Advanced Blackberry Development, Chapter 2 catpure, chapter 3 playback. Android – http://developer.android.com/intl/zh-CN/guide/topics/media/index.html http://developer.android.com/intl/zh-CN/guide/topics/media/index.html – http://www.brighthub.com/mobile/google-android/articles/43414.aspx (a difficult example to follow and it’s for 1.6) http://www.brighthub.com/mobile/google-android/articles/43414.aspx – Apress - Pro Android Media Developing Graphics, Music, Video, and Rich Media Apps for Smartphones and Tablets Chapter 2 for taking pictures, chapter 9 for video playback. – API examples, com.example.android.apis.media

57. Q A &