Presentation is loading. Please wait.

Presentation is loading. Please wait.

GStreamer T-111.5350 Multimedia programming Mika Ristimäki, Jenny Lahti.

Similar presentations


Presentation on theme: "GStreamer T-111.5350 Multimedia programming Mika Ristimäki, Jenny Lahti."— Presentation transcript:

1 GStreamer T-111.5350 Multimedia programming Mika Ristimäki, Jenny Lahti

2 History  The project was founded in 1999 by Erik Walthinsen.  Later it was joined by the maintainer of GStreamer, Wim Taymans.  The first corporate sponsor was RidgeRun, that hired Erik Walthinsen to develop methods for embedding GStreamer in smaller devices  The GStreamer logo was designed by Brock A. Frazier, who worked for RidgeRun.

3 Introduction  GStreamer was created to serve as a host for multimedia applications.  It is written in the C programming language  It is a cross-platform product, so it works e.g. on Linux, Solaris, Mac OS X and Microsoft Windows.  It provides a framework of plugins, for data flow and media type handling.  It is created to handle both audio and video.  With GStreamer’s development framework you can write any type of streaming multimedia application.

4 License – LGPL  GStreamer is released under LGPL (GNU Lesser General Public License).  LGPL means that the GStremer software itself has copyleft rules which means that you have to keep the software free.  But you are allowed to create your own software with your own copyright rules and link it with GStremer.  For companies like Nokia this license gives a profitable basis for their own software. They can use GStremer as a multimedia framework, but own the copyright to their own software, because it is merely linked to GStreamer.

5 Applications  GStreamer’s most obvious use is creating a media player on top of it, because it supports many needed formats for this.  GStreamer can also be used to create more complex programs like video or audio editing applications.  We intend to use it to create an Internet radio.

6 N770, N800 and N810  Nokia’s open source product N770 Internet Tablet and its successors N800 and N810, are palm-sized tablet PC:s running Linux  The tablets use GStreamer as a multimedia framework  This enables 3 rd party developers to use on-board DSP without extensive knowledge about its architecture  They support audio (MP3, MPEG4-AAC, WAV, AMR and MP2) and video (MPEG-1, MPEG-4, Real Video, H.263, AVI and 3GP).  They have a Real player and a Flash player

7 N770, N800 and N810

8 Lifetrak  A project at University of California, Department for Electrical Engineering by Sasank Reddy and Jeff Mascia.  Lifetrak uses Nokia 770 as a hardware platform for creating a portable music player.  Lifetrak enables a context-aware playlist that automatically selects music based upon the users location, the pace of his movement and the environment  Lifetrak has a simple learning mechanism that can adjust the selection of the music based on the user’s ratings.

9 Lifetrak

10  Lifetrak’s playing functionality is built on top of GStreamer  Lifetrak supports song files in the MP3 format, and in the future also Ogg-Vorbis, WAV, AAC, and MP4 files. GStreamer can handle each of these compression standards  GStreamer is also used for setting the volume, getting the position, and setting the position of the current song.

11 Mad Hatter’s Cocktail Party  A project at Standford University, Palo Alto Research Center and Robotics Institute in Carnegie Mellon University  Project wants to enhance conference call type mobile communication  The idea is to make communication possible for several small groups in a single audio space and for these groups to be able to regroup when desired.  The communication model is similar to a cocktail party.

12 Mad Hatter’s Cocktail Party  The idea is to analyse the audio, find these groups and make each listener hear his own group with a higher sound level and dimm out the voices of the rest.  This simulates a situation where you stand closer to the ones you listen to.  This is done by analysing when people speak, assuming that people in the same group mainly speek in turns.

13 Mad Hatter’s Cocktail Party  The server, that analyses the voices, is a PC running Linux using GStreamer.  The server receives audio streame from each client, analyzes them, applyes audio effect and performs mixing, and finally transmits audio back to the listeners.

14 Usage  GStreamer is so called “pipelined” multimedia framework  Pipe-line consists of sequence of operations  The basic building block of a pipeline is called an element  Pipe-line consists of chain of these elements and data flows through it.  For example, a basic mp3 audio decoder  Element 1 Reads mp3 file from hard disk.  Element 2 Decodes the mp3 data to raw audio  Element 3 Sends the raw audio to soundcard

15 Usage  Every element may also have a different number of pads  Sink (input) pad and source (output) pad.  Pad is an element’s link to outside world  Pads can also be dynamic so that a pad can be randomly created and/or destroyed.  Interconnected elements form a bin  Is also element  Same operations that can be done to an element can be done to a bin also  Pipeline is a specialized type of bin so that the top level bin has to be pipeline

16 Usage  A bus takes care of internal messaging in GStreamer.  A callback function can be defined that takes care of EOS or other errors messages and acts upon them.  Every pipe-line has a bus by default  The developer should create a callback function for the bus.  When the main loop is running the bus is checked for messages and when a new message is noticed the callback function will be called

17 Example #include int main (int argc, char *argv[]) { pipeline = gst_pipeline_new ("audio-player"); source = gst_element_factory_make ("filesrc", "file-source"); parser = gst_element_factory_make ("oggdemux", "ogg- parser"); decoder = gst_element_factory_make ("vorbisdec", "vorbis- decoder"); conv = gst_element_factory_make ("audioconvert", "converter"); sink = gst_element_factory_make ("alsasink", "alsa-output"); g_object_set (G_OBJECT (source), "location", argv[1], NULL); bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline)); gst_bus_add_watch (bus, bus_call, loop); gst_object_unref (bus); gst_bin_add_many (GST_BIN (pipeline), source, parser, decoder, conv, sink, NULL); gst_element_link (source, parser); gst_element_link_many (decoder, conv, sink, NULL); g_signal_connect (parser, "pad-added", G_CALLBACK (new_pad), NULL); }

18 Example #include int main (int argc, char *argv[]) { pipeline = gst_pipeline_new ("audio-player"); source = gst_element_factory_make ("filesrc", "file-source"); parser = gst_element_factory_make ("oggdemux", "ogg- parser"); decoder = gst_element_factory_make ("vorbisdec", "vorbis- decoder"); conv = gst_element_factory_make ("audioconvert", "converter"); sink = gst_element_factory_make ("alsasink", "alsa-output"); g_object_set (G_OBJECT (source), "location", argv[1], NULL); bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline)); gst_bus_add_watch (bus, bus_call, loop); gst_object_unref (bus); gst_bin_add_many (GST_BIN (pipeline), source, parser, decoder, conv, sink, NULL); gst_element_link (source, parser); gst_element_link_many (decoder, conv, sink, NULL); g_signal_connect (parser, "pad-added", G_CALLBACK (new_pad), NULL); } Creating elements

19 Example #include int main (int argc, char *argv[]) { pipeline = gst_pipeline_new ("audio-player"); source = gst_element_factory_make ("filesrc", "file-source"); parser = gst_element_factory_make ("oggdemux", "ogg- parser"); decoder = gst_element_factory_make ("vorbisdec", "vorbis- decoder"); conv = gst_element_factory_make ("audioconvert", "converter"); sink = gst_element_factory_make ("alsasink", "alsa-output"); g_object_set (G_OBJECT (source), "location", argv[1], NULL); bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline)); gst_bus_add_watch (bus, bus_call, loop); gst_object_unref (bus); gst_bin_add_many (GST_BIN (pipeline), source, parser, decoder, conv, sink, NULL); gst_element_link (source, parser); gst_element_link_many (decoder, conv, sink, NULL); g_signal_connect (parser, "pad-added", G_CALLBACK (new_pad), NULL); } Creating elements Setting filesrc element properties

20 Example #include int main (int argc, char *argv[]) { pipeline = gst_pipeline_new ("audio-player"); source = gst_element_factory_make ("filesrc", "file-source"); parser = gst_element_factory_make ("oggdemux", "ogg- parser"); decoder = gst_element_factory_make ("vorbisdec", "vorbis- decoder"); conv = gst_element_factory_make ("audioconvert", "converter"); sink = gst_element_factory_make ("alsasink", "alsa-output"); g_object_set (G_OBJECT (source), "location", argv[1], NULL); bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline)); gst_bus_add_watch (bus, bus_call, loop); gst_object_unref (bus); gst_bin_add_many (GST_BIN (pipeline), source, parser, decoder, conv, sink, NULL); gst_element_link (source, parser); gst_element_link_many (decoder, conv, sink, NULL); g_signal_connect (parser, "pad-added", G_CALLBACK (new_pad), NULL); } Creating elements Setting filesrc element properties Adding the callback function for bus

21 Example #include int main (int argc, char *argv[]) { pipeline = gst_pipeline_new ("audio-player"); source = gst_element_factory_make ("filesrc", "file-source"); parser = gst_element_factory_make ("oggdemux", "ogg- parser"); decoder = gst_element_factory_make ("vorbisdec", "vorbis- decoder"); conv = gst_element_factory_make ("audioconvert", "converter"); sink = gst_element_factory_make ("alsasink", "alsa-output"); g_object_set (G_OBJECT (source), "location", argv[1], NULL); bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline)); gst_bus_add_watch (bus, bus_call, loop); gst_object_unref (bus); gst_bin_add_many (GST_BIN (pipeline), source, parser, decoder, conv, sink, NULL); gst_element_link (source, parser); gst_element_link_many (decoder, conv, sink, NULL); g_signal_connect (parser, "pad-added", G_CALLBACK (new_pad), NULL); } Adding the callback function for bus Putting elements to a bin Creating elements Setting filesrc element properties

22 Example #include int main (int argc, char *argv[]) { pipeline = gst_pipeline_new ("audio-player"); source = gst_element_factory_make ("filesrc", "file-source"); parser = gst_element_factory_make ("oggdemux", "ogg- parser"); decoder = gst_element_factory_make ("vorbisdec", "vorbis- decoder"); conv = gst_element_factory_make ("audioconvert", "converter"); sink = gst_element_factory_make ("alsasink", "alsa-output"); g_object_set (G_OBJECT (source), "location", argv[1], NULL); bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline)); gst_bus_add_watch (bus, bus_call, loop); gst_object_unref (bus); gst_bin_add_many (GST_BIN (pipeline), source, parser, decoder, conv, sink, NULL); gst_element_link (source, parser); gst_element_link_many (decoder, conv, sink, NULL); g_signal_connect (parser, "pad-added", G_CALLBACK (new_pad), NULL); } Putting elements to a bin Creating elements Setting filesrc element properties Linking elements together Adding the callback function for bus

23 Example static gboolean bus_call (GstBus *bus, GstMessage *msg, gpointer data) { GMainLoop *loop = (GMainLoop *) data; switch (GST_MESSAGE_TYPE (msg)) { case GST_MESSAGE_EOS: g_print ("End-of-stream\n"); g_main_loop_quit (loop); break; case GST_MESSAGE_ERROR: { gchar *debug; GError *err; gst_message_parse_error (msg, &err, &debug); g_free (debug); g_print ("Error: %s\n", err->message); g_error_free (err); g_main_loop_quit (loop); break; } default: break; } return TRUE; } static void new_pad (GstElement *element, GstPad *pad, gpointer data) { GstPad *sinkpad; /* We can now link this pad with the audio decoder */ g_print ("Dynamic pad created, linking parser/decoder\n"); sinkpad = gst_element_get_pad (decoder, "sink"); gst_pad_link (pad, sinkpad); gst_object_unref (sinkpad); } Callback funtion for error handling Function for dynamic pad creation

24 References  [1] GStreamer Application Development Manual (0.10.12.1), Wim Taymans, Steve Baker, Andy Wingo, Ronald S. Bultje, Stefan Kost  [2] DSP video processing via open-source APIs, EETimes OnLine, Rishi Bhatteracharya, 30.10.2006 http://www.eetimes.com/news/design/showArticle.jhtml?articleID=193401461 http://www.eetimes.com/news/design/showArticle.jhtml?articleID=193401461  [3] The Mad Hatter’s Coktail Party: A Social Mobile Audio Space Supporting Multiple Simultaneous Conversations, Paul M. Aoki, Matthew Romaine, Margaret H. Szymanski, James D. Thornton, Daniel Wilson and Allison Woodruff, Palo Alto Research Center, CCRMA, Dept. of Music Stanford University, Robotics Institute Carnegie Mellon University, Florida USA 5.-10.43.2003, Paper: Design for the Socially Mobile  [4] Lifetrak: Music In Tune With Your Life, Sasank Reddy, Jeff Mascia, NESL, Department of Electrical Engineering University of California, Los Angeles  [5] Multimedia Abstraction Architecture of Cross-platform Applications Based on IMS, Jie Chen, Master Thesis, Helsinki University of Technology / Department of Computer Science and Engineering, 2006  [6] Linux for suits: a first look at the Nokia 770, Doc Searls, 02/2006, Linux Journal, Volume 2006 Issue 142, Publisher: Specialized Systems Consultants, Inc.  [7] Wikipedia, The Free Encyclopedia, (GStreamer, LGPL), http://en.wikipedia.org/wiki/Gstreamer, http://en.wikipedia.org/wiki/GNU_Lesser_General_Public_Licensehttp://en.wikipedia.org/wiki/Gstreamer http://en.wikipedia.org/wiki/GNU_Lesser_General_Public_License


Download ppt "GStreamer T-111.5350 Multimedia programming Mika Ristimäki, Jenny Lahti."

Similar presentations


Ads by Google