Jonathan Beluch2012-09-17T22:12:00Zhttp://jonathanbeluch.com/Automated Testing of XBMC AddonsJonathan Beluch2012-09-17T17:48:00Z2012-09-17T17:48:00Zhttp://jonathanbeluch.com/blog/2012/09/testing-xbmc-addons/
<p>If you are not familiar with <span class="caps"><span class="caps">XBMC</span></span> add-ons, they are python scripts that
interface between a website and <span class="caps"><span class="caps">XBMC</span></span>. They either interact with a website <span class="caps"><span class="caps">API</span></span>
or they scrape <span class="caps"><span class="caps">HTML</span></span> pages and then pass the video content to <span class="caps"><span class="caps">XBMC</span></span> via its python bindings.</p>
<p>Since most add-ons are basically website scrapers, this means they tend to
break somewhat often. Currently, when an add-on breaks I’m notified in a
multitude of ways including email, forum posts and private messages, github
issues and mailing list emails. This presents a problem as I sometimes find
out my add-on is broken a week or two late!</p>
<p>The answer to this problem is automated testing. This post will cover basic
examples of <a href="http://en.wikipedia.org/wiki/Unit_testing">unit testing</a> and <a href="http://en.wikipedia.org/wiki/Integration_testing">integration testing</a>. Our unit tests will
run every time we make a commit, but will not interact with any external APIs
or websites. Our integration tests will run every commit but also on a daily
schedule. The purpose of the <span class="caps"><span class="caps">IT</span></span> tests is to verify that our code works properly
with the website or <span class="caps"><span class="caps">API</span></span> (and therefore that the remote source has not changed).</p>
<h3 id="setup">Setup</h3>
<p>This post assumes you have a basic knowledge of developing <span class="caps"><span class="caps">XBMC</span></span> add-ons. We’ll
use my <a href="https://github.com/jbeluch/xbmc-vimcasts">xbmc-vimcasts</a> add-on as an example in this post. This add-on uses my
<a href="http://xbmcswift.com">xbmcswift2</a> framework, as it facilitates easier command line execution and
testing. To run our unit and <span class="caps"><span class="caps">IT</span></span> tests in an automated fashion, we’re going to
use <a href="http://travis-ci.org">Travis <span class="caps"><span class="caps">CI</span></span></a> since they offer simple <a href="https://github.com">github</a> integration.</p>
<h3 id="writing-our-unit-tests">Writing our unit tests</h3>
<p>I like to keep my add-on tests in <code>resources/tests</code>. I also use <code>python -m
unittest discover</code> to run the tests, so make sure to create
<code>resources/tests/__init__.py</code> as well. Let’s create our first test file,
<code>resources/tests/test_addon.py</code>.</p>
<div class="codehilite"><pre><span class="kn">import</span> <span class="nn">sys</span><span class="o">,</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">unittest</span>
<span class="n">sys</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">__file__</span><span class="p">),</span> <span class="s">'../..'</span><span class="p">))</span>
<span class="kn">from</span> <span class="nn">addon</span> <span class="kn">import</span> <span class="n">plugin</span><span class="p">,</span> <span class="n">strip_tags</span><span class="p">,</span> <span class="n">unescape_html</span><span class="p">,</span> <span class="n">clean</span><span class="p">,</span> <span class="n">get_json_feed</span><span class="p">,</span> <span class="n">index</span>
</pre></div>
<p>Since <span class="caps"><span class="caps">XBMC</span></span> add-ons aren’t installed python packages, we need to do some path
trickery in order to import our add-on as a module (This is why xbmcswift2
recommends using the <code>if __name__ == '__main__'</code> guard in your addon.py).</p>
<p>There are a few basic functions in addon.py that don’t go out to the remote
<span class="caps"><span class="caps">API</span></span>. These are perfect for unit tests.</p>
<p>We’ll create a class to hold all of our unit tests:</p>
<div class="codehilite"><pre><span class="k">class</span> <span class="nc">TestNonViews</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">test_strip_tags</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">known_values</span> <span class="o">=</span> <span class="p">[</span>
<span class="p">(</span><span class="s">'<b>Hello</b>'</span><span class="p">,</span> <span class="s">'Hello'</span><span class="p">),</span>
<span class="p">(</span><span class="s">'Hello'</span><span class="p">,</span> <span class="s">'Hello'</span><span class="p">),</span>
<span class="p">(</span><span class="s">'<b>'</span><span class="p">,</span> <span class="s">''</span><span class="p">),</span>
<span class="p">(</span><span class="s">'<b><a href="#">Hello</a></b>'</span><span class="p">,</span> <span class="s">'Hello'</span><span class="p">),</span>
<span class="p">]</span>
<span class="k">for</span> <span class="n">inp</span><span class="p">,</span> <span class="n">expected</span> <span class="ow">in</span> <span class="n">known_values</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">strip_tags</span><span class="p">(</span><span class="n">inp</span><span class="p">),</span> <span class="n">expected</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_unescape_html</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">known_values</span> <span class="o">=</span> <span class="p">[</span>
<span class="p">(</span><span class="s">'&gt;jon&dave'</span><span class="p">,</span> <span class="s">'>jon&dave'</span><span class="p">),</span>
<span class="p">]</span>
<span class="k">for</span> <span class="n">inp</span><span class="p">,</span> <span class="n">expected</span> <span class="ow">in</span> <span class="n">known_values</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">unescape_html</span><span class="p">(</span><span class="n">inp</span><span class="p">),</span> <span class="n">expected</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_clean</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">known_values</span> <span class="o">=</span> <span class="p">[</span>
<span class="p">(</span><span class="s">'<b>jon &amp; dave</b>'</span><span class="p">,</span> <span class="s">'jon <span class="amp">&</span> dave'</span><span class="p">),</span>
<span class="p">]</span>
<span class="k">for</span> <span class="n">inp</span><span class="p">,</span> <span class="n">expected</span> <span class="ow">in</span> <span class="n">known_values</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">clean</span><span class="p">(</span><span class="n">inp</span><span class="p">),</span> <span class="n">expected</span><span class="p">)</span>
</pre></div>
<p>At the bottom of our test file, we need to include the call to execute the tests:</p>
<div class="codehilite"><pre><span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">unittest</span><span class="o">.</span><span class="n">main</span><span class="p">()</span>
</pre></div>
<p>Now, let’s run our tests using <code>python -m unittest discover</code>. You should see
output similar to the following:</p>
<div class="codehilite"><pre><span class="p">(</span><span class="n">xbmc</span><span class="o">-</span><span class="n">vimcasts</span><span class="p">)</span><span class="n">jon</span><span class="p">@</span><span class="n">lenovo</span> <span class="o">~/</span><span class="n">Code</span><span class="o">/</span><span class="n">xbmc</span><span class="o">-</span><span class="n">vimcasts</span> <span class="p">(</span><span class="n">master</span><span class="p">)</span> $ <span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">unittest</span> <span class="n">discover</span>
<span class="p">.....</span>
<span class="o">----------------------------------------------------------------------</span>
<span class="n">Ran</span> 5 <span class="n">tests</span> <span class="n">in</span> 2<span class="p">.</span>951<span class="n">s</span>
<span class="n"><span class="caps"><span class="caps">OK</span></span></span>
</pre></div>
<p>Now that we have working unit tests, we should make it a habit to always run
the test suite before committing any changes.</p>
<h3 id="writing-our-it-tests">Writing our <span class="caps"><span class="caps">IT</span></span> tests</h3>
<p>Our <span class="caps"><span class="caps">IT</span></span> tests will be very similar looking to our unit tests. The main
difference is that these tests will be crossing the network boundary, and the
scope of the test will be much larger than a single function or “unit”. In
order to test that our code handles the <span class="caps"><span class="caps">API</span></span> correctly, we basically end up
testing the remote <span class="caps"><span class="caps">API</span></span> as well!</p>
<div class="codehilite"><pre><span class="p">:::</span> <span class="n">python</span>
<span class="k">class</span> <span class="nc">ITTests</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">test_api</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">resp</span> <span class="o">=</span> <span class="n">get_json_feed</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertTrue</span><span class="p">(</span><span class="s">'episodes'</span> <span class="ow">in</span> <span class="n">resp</span><span class="o">.</span><span class="n">keys</span><span class="p">())</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertTrue</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">resp</span><span class="p">[</span><span class="s">'episodes'</span><span class="p">])</span> <span class="o">></span> <span class="mi">35</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_index</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">items</span> <span class="o">=</span> <span class="n">index</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertTrue</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">items</span><span class="p">)</span> <span class="o">></span> <span class="mi">35</span><span class="p">)</span>
<span class="n">expected</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'info'</span><span class="p">:</span> <span class="p">{</span>
<span class="s">'plot'</span><span class="p">:</span> <span class="s">u'Vim</span><span class="se">\u2019</span><span class="s">s list feature can be used to reveal hidden characters, such as tabstops and newlines. In this episode, I demonstrate how to customise the appearance of these characters by tweaking the listchars setting. I go on to show how to make these invisible characters blend in with your colortheme.</span><span class="se">\n</span><span class="s">'</span><span class="p">},</span>
<span class="s">'is_playable'</span><span class="p">:</span> <span class="bp">True</span><span class="p">,</span>
<span class="s">'label'</span><span class="p">:</span> <span class="s">u'#1 Show invisibles'</span><span class="p">,</span>
<span class="s">'path'</span><span class="p">:</span> <span class="s">u'http://media.vimcasts.org/videos/1/show_invisibles.m4v'</span><span class="p">,</span>
<span class="s">'thumbnail'</span><span class="p">:</span> <span class="s">u'http://vimcasts.org/images/posters/show_invisibles.png'</span>
<span class="p">}</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">items</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">expected</span><span class="p">)</span>
</pre></div>
<p>For the assert statements in our <span class="caps"><span class="caps">IT</span></span> tests, we must find an equilibrium in how
specific our tests are. We don’t want to have to update our tests if a website
posts a new video. But, we would also like to be notified if they change the
format of the website. If we are scraping content that is always changing, it
will be hard to match exact string values and URLs. Typically, if I am
expecting a list of items as a response, I try to assert using <code>></code> or <code><</code>
rather than <code>==</code> when testing the number of items returned. For instance, there
are about 36 videos at the time of writing this post. So I want to verify that
there are at least 35 items in the response which leaves a little wiggle room.
If more videos are added, I don’t want my test to fail, so I won’t test for an
exact number of items. If there are 0 or 1 videos, something is obviously
wrong and the test will fail. However, if just 1 video is removed for some
reasons, the test should still pass.</p>
<p>In the above example, I expect the order of the returned items to always be
sorted, so I can verify one of the items from the list.</p>
<h3 id="travis-integration">Travis integration</h3>
<p>If you use <a href="https://github.com">github</a> for your add-on, you have the ability to use <a href="http://travis-ci.org">Travis
<span class="caps"><span class="caps">CI</span></span></a> to run your test suite after every commit. See <a href="http://about.travis-ci.org/docs/user/getting-started/">getting started</a> for
instructions on setting up your repository for testing.</p>
<p>The simplest .travis.yml file for our add-on is:</p>
<div class="codehilite"><pre><span class="l-Scalar-Plain">language</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">python</span>
<span class="l-Scalar-Plain">python</span><span class="p-Indicator">:</span>
<span class="p-Indicator">-</span> <span class="s">"2.7"</span>
<span class="c1"># command to install dependencies, e.g. pip install -r requirements.txt --use-mirrors</span>
<span class="l-Scalar-Plain">install</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">pip install --use-mirrors xbmcswift2==0.1 beautifulsoup</span>
<span class="c1"># command to run tests, e.g. python setup.py test</span>
<span class="l-Scalar-Plain">script</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">python -m unittest discover</span>
</pre></div>
<p>Once you’ve enabled Travis, you can go ahead and commit the .travis.yml to your
repository. If everything goes well your tests should execute automatically
(and hopefully pass).</p>
<h3 id="running-tests-daily">Running tests daily</h3>
<p>Now we have the ability to automatically run our unit/<span class="caps"><span class="caps">IT</span></span> tests after every
commit. Our unit tests only need to be run before each commit, once the code
works, it works for good until changes are made. However, to be effective, our
<span class="caps"><span class="caps">IT</span></span> tests should run at least once a day.</p>
<p>There currently is not a scheduling feature for Travis <span class="caps"><span class="caps">CI</span></span>, but with a little
work we can get close. If you go to your repository admin page on github and
select the Travis hook, you’ll notice a <em>test hook</em> button. If you click this
button, your Travis tests will be run with the current branch’s latest commit.
The <em>test hook</em> button can be activated via the github <span class="caps"><span class="caps">API</span></span>. So, we’re going to
set up a cron job that will activate the hook once a day.</p>
<p>I wrote a simple script, <a href="https://github.com/jbeluch/githubhooks">githubhooks</a>, that allows you to list and test your
activated github hooks. You can install it with <code>pip install githubhooks</code>.
You’ll also need your github OAuth token, which can be created by following
<a href="https://help.github.com/articles/creating-an-oauth-token-for-command-line-use">these directions</a>.</p>
<p>Once you have your token, you can set it as an environment variable
<code>GITHUB_TOKEN</code>, or pass it as an argument to <code>githubhooks.py</code>. Each repository
hook has an <em>id</em>, which we’ll use to test the hook. To list our hook ids by
repository, we’ll run the following:</p>
<div class="codehilite"><pre><span class="p">(</span><span class="n">github</span><span class="o">-</span><span class="n">hooks</span><span class="p">)</span><span class="n">jon</span><span class="p">@</span><span class="n">lenovo</span> <span class="o">~/</span><span class="n">Code</span><span class="o">/</span><span class="n">github</span><span class="o">-</span><span class="n">hooks</span> $ <span class="n">githubhooks</span><span class="p">.</span><span class="n">py</span> <span class="n">list</span>
<span class="p">..</span> <span class="n">Listing</span> <span class="n">Hooks</span> <span class="n">by</span> <span class="n">Repository</span> <span class="p">..</span>
<span class="n">xam</span><span class="p">:</span>414859 <span class="p">(</span><span class="n">travis</span><span class="p">)</span>
<span class="n">xbmc</span><span class="o">-</span><span class="n">vimcasts</span><span class="p">:</span>416593 <span class="p">(</span><span class="n">travis</span><span class="p">)</span>
<span class="n">xbmcswift</span><span class="p">:</span>78826 <span class="p">(</span><span class="n">twitter</span><span class="p">)</span>
<span class="n">xbmcswift2</span><span class="p">:</span>239826 <span class="p">(</span><span class="n">readthedocs</span><span class="p">)</span>
</pre></div>
<p>Now, let’s kick off a hook:</p>
<div class="codehilite"><pre><span class="p">(</span><span class="n">github</span><span class="o">-</span><span class="n">hooks</span><span class="p">)</span><span class="n">jon</span><span class="p">@</span><span class="n">lenovo</span> <span class="o">~/</span><span class="n">Code</span><span class="o">/</span><span class="n">github</span><span class="o">-</span><span class="n">hooks</span> $ <span class="n">githubhooks</span><span class="p">.</span><span class="n">py</span> <span class="o">--</span><span class="n">hook</span> <span class="n">xbmc</span><span class="o">-</span><span class="n">vimcasts</span><span class="p">:</span>416593 <span class="n">run</span>
<span class="n">Triggering</span> <span class="n">hook</span> <span class="n">travis</span> <span class="k">for</span> <span class="n">xbmc</span><span class="o">-</span><span class="n">vimcasts</span><span class="p">...</span> <span class="n"><span class="caps"><span class="caps">OK</span></span></span>
</pre></div>
<p>You can verify on Travis <span class="caps"><span class="caps">CI</span></span> that your tests were executed. The final step is
run the test command via cron every 24 hours. If your tests fail for any
reason, Travis will send you an email. Now you can rest easy, knowing you’ll be
notified in less than 24 hours if your add-on breaks (assuming you’ve written good tests)!</p>
Auto-archive emails in gmailJonathan Beluch2011-04-09T15:57:00Z2011-04-09T15:57:00Zhttp://jonathanbeluch.com/blog/2011/04/gmail-auto-archiver/
<p>If you’re like me, you futilely attempt to adhere to <em>inbox zero</em>. Subscribing
to a lot of daily emails makes it much harder to keep a clean inbox. I prefer
to read these emails, so I don’t have them skip my inbox. However, after 2 or 3
days, I’m not going to read the email if I haven’t already. </p>
<p>Since gmail doesn’t let you auto-archive after a certain date, I made this
<a href="https://github.com/jbeluch/gmail-autoarchiver">python script</a> that does just that. You simply set up labels in gmail
like this <code>autoarchive:3</code> where <code>3</code> is the age limit in days.</p>
<p>Then simply enter your credentials in the python script, or enter them
interactively when the script is run.</p>
<p>The script logs in and gets all your labels similar to the pattern
<code>autoarchive:*</code>. Then for each label, it downloads all associated emails that
are in your <span class="caps"><span class="caps">INBOX</span></span>. It checks the <code>Date</code> header of the email (This can be forged
obviously, maybe I’ll check the Received header in the future). If the date is
older than the specified age in days, it adds an <span class="caps"><span class="caps">IMAP</span></span> flag of <code>\DELETED</code>. In
gmail this is how you archive an email, it does not actually <em>delete</em> it.</p>
<p><a href="https://github.com/jbeluch/gmail-autoarchiver">Check it out on github</a></p>
Remote notifications using screen, irssi, and lib-notify.Jonathan Beluch2011-03-31T20:14:00Z2011-03-31T20:14:00Zhttp://jonathanbeluch.com/blog/2011/03/remote-notify-irssi-screen/
<p>I hate having multiple <span class="caps"><span class="caps">IM</span></span> windows open. I usually have 3 or 4 accounts I like
to log into in addition to irc. The solution to this madness is <a href="http://www.bitlbee.org">bitlbee</a>.
It’s an <span class="caps"><span class="caps">IM</span></span> gateway that supports many <span class="caps"><span class="caps">IM</span></span> networks and protocols and brings them
all into your irc client of choice.</p>
<p>My setup is to run bitlbee on a remote server. I use <a href="http://www.irssi.org">irssi</a> as my irc
client, and I run it inside a <a href="http://www.gnu.org/software/screen/">screen</a> session. When I want to chat, I <span class="caps"><span class="caps">SSH</span></span>
into the remote server, attach to the running irssi screen session and login to
my <span class="caps"><span class="caps">IM</span></span> accounts.</p>
<p>Since I’m always using multiple workspaces with my xmonad-gnome setup, I
sometimes forget to check my irssi window to see if someone has messaged me. I
wanted to use <code>notify-send</code> to be able to display a notification when someone
messaged me. As it turns out, other people have already done all the hard work.</p>
<p>There are a few different solutions, but <a href="http://esaurito.net/blog/posts/2008/11/irssi_notify/">here</a> is the solution I found
easiest to work with. There are 3 files to deal with. The first is the script
for irssi, <code>rnotify.pl</code>, which lives on the remote machine. The other two
files, <code>notify-remote</code> and <code>irssi-connect</code>, live on your local machine.</p>
<h2 id="setup">Setup</h2>
<p>Download the 3 necessary files from my github <a href="https://gist.github.com/898734">gist</a>. I’ve made some minor
changes to the two shell scripts found on <a href="http://esaurito.net/blog/posts/2008/11/irssi_notify/">esaurito’s</a> site.</p>
<h3 id="remote-machine">Remote machine</h3>
<p>Follow these steps on the remote machine (where you run screen, irssi).</p>
<ol>
<li>Place <code>rnotify.pl</code> in <code>~/.irssi/scripts/</code>.</li>
<li>Create a symlink in <code>~/.irssi/scripts/autorun/</code> to the rnotify script</li>
<li>If you already have irssi started, you’ll need to manually load the script
this one time. In irssi, <code>/script load rnotify.pl</code> </li>
</ol>
<h3 id="local">Local</h3>
<p>The shell script requires, <code>autossh</code> and <code>socat</code>.</p>
<ol>
<li>
<p>You’ll need to modify <code>irssi-connect</code> to change a few config options. You’ll
have to edit your <code>host</code> to reflect your remote server’s hostname. </p>
</li>
<li>
<p>You will probably have to edit the <code>notify</code> line as well, to reflect the
path to the other file, <code>notify-remote</code>.</p>
</li>
<li>
<p>If you ever have more than 1 screen session running on your remote server,
you’ll have to modify the autossh line to reflect the name of the session.</p>
</li>
</ol>
<h2 id="usage">Usage</h2>
<p>Simply execute <code>./irssi-connect</code> and the magic should happen. If you are
somewhere that blocks outgoing connections to port 22, you can pass in an
alternate port for <span class="caps"><span class="caps">SSH</span></span>, e.g. 443 (you do listen on 443 as well, don’t you?).</p>
<p>The final step is to add an alias to your shell (<code>~/.bashrc</code> in my case). I use
<code>alias irc='/home/jon/scripts/irssi-rnotify/fedora-connect'</code>. Now when you
want to use irc with notifications, you should simply be able to call <code>irc</code> or
<code>irc 443</code>.</p>
<h3 id="modiftying-the-notification-timeout">Modiftying the Notification Timeout</h3>
<p>The script uses notify-send which uses notify-osd to display the notifications.
The default timeout is 5 seconds which is way too long, especially if you get
multiple messages in a row. Curiously, the man page for notify-send lists a
<code>-t, --expire-time=TIME</code> option, which should let you specify the timeout.
However, <a href="https://bugs.launchpad.net/ubuntu/+source/notify-osd/+bug/390508">it is ignored!</a>.<br />
</p>
<p>If you are on Ubuntu 10.04, you can add <a href="http://www.webupd8.org/2010/07/patched-notifyosd-updates-option-to.html">Sukochev
Roman’s private repository</a>, which contains a modified version of
notify-osd. This will allow you to specify the timeout in the notify-remote
script. I use 2000 ms.</p>
RTMP SniffingJonathan Beluch2011-01-19T17:47:00Z2011-01-19T17:47:00Zhttp://jonathanbeluch.com/blog/2011/01/rtmp-sniffing/
<p>This post will outline a quick and easy method for gathering info about an
<a href="http://en.wikipedia.org/wiki/Real_Time_Messaging_Protocol"><span class="caps"><span class="caps">RTMP</span></span></a> stream. The goal is to gather the proper parameters to enable
playback of the <span class="caps"><span class="caps">RTMP</span></span> stream in <a href="http://xbmc.org/"><span class="caps"><span class="caps">XBMC</span></span></a>.</p>
<p>This tutorial will utilize two command lines tools that should be available
in most package managers.</p>
<ol>
<li><a href="http://www.tcpdump.org/">tcpdump</a> - A network capture tool</li>
<li><a href="http://rtmpdump.mplayerhq.hu/">rtmpdump</a> - An <span class="caps"><span class="caps">RTMP</span></span> streaming media client. We will be using this to
verify our <span class="caps"><span class="caps">RTMP</span></span> parameters.</li>
</ol>
<p>To gather the required <span class="caps"><span class="caps">RTMP</span></span> parameters, we are going to capture the network
traffic between our local machine and the <span class="caps"><span class="caps">RTMP</span></span> server while playing a target
video in a browser.</p>
<div class="toc">
<ul>
<li><a href="#overview">Overview</a></li>
<li><a href="#rtmp-parameters"><span class="caps"><span class="caps">RTMP</span></span> Parameters</a></li>
<li><a href="#capturing-the-stream">Capturing the Stream</a><ul>
<li><a href="#setting-up-the-capture">Setting up the capture</a></li>
<li><a href="#tcpdump">tcpdump</a></li>
</ul>
</li>
<li><a href="#searching-the-packet-capture">Searching the packet capture</a><ul>
<li><a href="#finding-connect-parameters">Finding connect parameters</a></li>
<li><a href="#finding-play-parameters">Finding play parameters</a></li>
</ul>
</li>
<li><a href="#testing-the-captured-rtmp-parameters">Testing the captured <span class="caps"><span class="caps">RTMP</span></span> parameters</a></li>
<li><a href="#playing-an-rtmp-video-in-xbmc">Playing an <span class="caps"><span class="caps">RTMP</span></span> video in <span class="caps"><span class="caps">XBMC</span></span></a></li>
</ul>
</div>
<h2 id="overview">Overview</h2>
<p>The <a href="http://www.adobe.com/devnet/rtmp.html"><span class="caps"><span class="caps">RTMP</span></span> protocol</a> is owned by Adobe and used for streaming media.
You typically encounter this technology on a website that uses a flash player.
The binary <a href="http://en.wikipedia.org/wiki/SWF">swf</a> file that your browser downloads contains all the information
needed to connect directly to the <span class="caps"><span class="caps">RTMP</span></span> server. </p>
<p>If you have an <span class="caps"><span class="caps">SWF</span></span> decompiler, you can decompile the swf and look through the
source for the proper parameters. However, I think it is both easier and faster
to sniff the actual network traffic being used by flash. Since flash makes the
connections to the <span class="caps"><span class="caps">RTMP</span></span> server outside of your browser, we cannot simply use
google chrome developer tools or something of that nature. We need to capture
packets from the <span class="caps"><span class="caps">OS</span></span>.</p>
<h2 id="rtmp-parameters"><span class="caps"><span class="caps">RTMP</span></span> Parameters</h2>
<p>There are lots of possible <span class="caps"><span class="caps">RTMP</span></span> connection parameters that can be set to play a
given video. We will be trying to find the minimum needed to successfully play
a video. We are going to be interested in:</p>
<ul>
<li>app</li>
<li>tcUrl</li>
<li>playpath</li>
</ul>
<p>For a complete list, check out the <a href="http://www.adobe.com/content/dam/Adobe/en/devnet/rtmp/pdf/rtmp_specification_1.0.pdf"><span class="caps"><span class="caps">RTMP</span></span> specification (pdf)</a> or the man page for
<em>rtmpdump</em>.</p>
<h2 id="capturing-the-stream">Capturing the Stream</h2>
<p>For our example, we are going to use <a href="">http://mitworld.mit.edu</a>. They have
plenty of awesome videos, and it’s also the website I used while learning this myself.</p>
<h3 id="setting-up-the-capture">Setting up the capture</h3>
<p>First we’ll need to figure out the hostname of the <span class="caps"><span class="caps">RTMP</span></span> server. If you
navigate to a <a href="http://mitworld.mit.edu/video/867">video page</a>, you can search through the source for
‘host:’. Or easier yet,
<code>curl -s "http://mitworld.mit.edu/video/867" | grep "host:"</code>. </p>
<p>We’re going to add this hostname as a filter to our network capture. We don’t
necessarily need to, but packet captures can grow very large depending on what
else is running on the machine. Your best bet is to filter the traffic as much as possible.</p>
<h3 id="tcpdump">tcpdump</h3>
<p>Here is our tcpdump command, which we can kick off.</p>
<div class="codehilite"><pre><span class="nv">$ </span>tcpdump -w mitworld.cap -s 0 -i wlan0 host cp58255.edgefcs.net
</pre></div>
<ul>
<li><code>-w mitworld.cap</code> - Write raw packets to a capture file.</li>
<li><code>-s 0</code> - Grab up 65,535 bytes from each packet. On some versions of tcpdump,
the default is too small to grab the entire packet.</li>
<li><code>-i wlan0</code> - The network interface to listen on, in my case my wireless card.</li>
<li><code>host cp58255.edgefcs.net</code> - Only match packets which are to/from a given host.</li>
</ul>
<p>After kicking it off (you might need root privileges) you should see output
similar to the following:</p>
<div class="codehilite"><pre>tcpdump: listening on wlan0, link-type EN10MB (Ethernet), capture size 65535 bytes
</pre></div>
<p>Now, visit the <a href="http://mitworld.mit.edu/video/867">video page</a> again. The parameters we are interested in will
be sent once we click play, so it’s only necessary to run the capture for a
second or so. Once the movie begins playing, you can halt the capture with
<em>Ctrl + C</em>.</p>
<p>When you halt the capture, you should see some output telling you how many
packets were capture. </p>
<h2 id="searching-the-packet-capture">Searching the packet capture</h2>
<p><span class="caps"><span class="caps">RTMP</span></span> encodes objects using the binary <a href="http://en.wikipedia.org/wiki/Action_Message_Format"><span class="caps"><span class="caps">AMF</span></span></a> protocol. There are two versions
of <span class="caps"><span class="caps">AMF</span></span>, <a href="http://opensource.adobe.com/wiki/download/attachments/1114283/amf0_spec_121207.pdf"><span class="caps"><span class="caps">AMF0</span></span> (pdf)</a> and <a href="http://opensource.adobe.com/wiki/download/attachments/1114283/amf3_spec_05_05_08.pdf"><span class="caps"><span class="caps">AMF3</span></span> (pdf)</a>. If you’d like to read more about the
protocol, the open-source project <a href="http://www.gnashdev.org/">gnash</a> has some good stuff in their
<a href="http://en.wikipedia.org/wiki/Gnash">wiki</a>, however it’s not necessary for this exercise.</p>
<p>There are 2 <span class="caps"><span class="caps">RTMP</span></span> command we are interested in, <strong>connect</strong> and <strong>play</strong>.</p>
<h3 id="finding-connect-parameters">Finding <em>connect</em> parameters</h3>
<p>We are going to output our packet capture with <span class="caps"><span class="caps">ASCII</span></span> characters (<code>-X</code>) and then
grep for our matching packets. You might need to mess with the numbers for grep
specifying <code>-B</code>efore context lines and <code>-A</code>fter context lines to grep.</p>
<div class="codehilite"><pre><span class="nv">$ </span>tcpdump -X -r mitworld.cap | grep -B 5 -A 20 connect
</pre></div>
<div class="codehilite"><pre>
0x00b0: 9b56 a0c1 9ee8 2764 dc3e 2e4b b188 350b .V....'d.>.K..5.
0x00c0: 802b 7d88 0d16 003d 44e7 e13d 04e9 a031 .+}....=D..=...1
0x00d0: c16f 8c41 363c a31b 3769 3ea6 9d4e 7b81 .o.A6<..7i>..N{.
0x00e0: a655 0567 f4b5 96ba 1a99 e29e 28d6 f7d0 .U.g........(...
0x00f0: eb92 3ef8 0300 0001 0001 8d14 0000 0000 ..>.............
0x0100: 0200 0763 6f6e 6e65 6374 003f f000 0000 ...connect.?....
0x0110: 0000 0003 0003 6170 70<span class="k">02</span> <span class="o">0027</span> <span class="s2">6f6e 6465</span> ......app<span class="k">.</span><span class="o">.'</span><span class="s2">onde</span>
0x0120: <span class="s2">6d61 6e64 3f5f 6663 735f 7668 6f73 743d mand?_fcs_vhost=</span>
0x0130: <span class="s2">6370 3538 3235 352e 6564 6765 6663 732e cp58255.edgefcs.</span>
0x0140: <span class="s2">6e65 74</span>00 0866 6c61 7368 5665 7202 000d <span class="s2">net</span>..flashVer...
0x0150: 4c4e 5820 3130 2c30 2c34 352c 3200 0673 <span class="caps"><span class="caps">LNX</span></span>.10,0,45,2..s
0x0160: 7766 5572 6c<span class="k">02</span> <span class="o">002d</span> <span class="s2">6874 7470 3a2f 2f6d</span> wfUrl<span class="k">.</span><span class="o">.-</span><span class="s2">http://m</span>
0x0170: <span class="s2">6974 776f 726c 642e 6d69 742e 6564 752f itworld.mit.edu/</span>
0x0180: <span class="s2">c366 6c61 7368 2f70 6c61 7965 722f 4d61 .flash/player/Ma</span>
0x0190: <span class="s2">696e 2e73 7766</span> 0005 7463 5572 6c<span class="k">02</span> <span class="o">0040</span> <span class="s2">in.swf</span>..tcUrl<span class="k">.</span><span class="o">.@</span>
0x01a0: <span class="s2">7274 6d70 3a2f 2f38 302e 3135 342e 3131 rtmp://80.154.11</span>
0x01b0: <span class="s2">382e 3232 3a34 3433 2f6f 6e64 656d 616e 8.22:443/ondeman</span>
0x01c0: <span class="s2">643f 5f66 6373 5f76 686f 7374 3d63 7035 d?_fcs_vhost=cp5</span>
0x01d0: <span class="s2">3832 3535 2e65 6467 6566 6373 2e6e 6574 8255.edgefcs.net</span>
0x01e0: 0004 6670 6164 0100 000c 6361 7061 6269 ..fpad....capabi
0x01f0: 6c69 7469 6573 0040 2e00 0000 0000 0000 lities.@........
0x0200: 0bc3 6175 6469 6f43 6f64 6563 7300 40a8 ..audioCodecs.@.
0x0210: ee00 0000 0000 000b 7669 6465 6f43 6f64 ........videoCod
0x0220: 6563 7300 406f 8000 0000 0000 000d 7669 ecs.@o........vi
0x0230: 6465 6f46 756e 6374 696f 6e00 3ff0 0000 deoFunction.?...
0x0240: 0000 0000 0007 7061 6765 5572 6c<span class="k">02</span> <span class="o">0021</span> ......pageUrl<span class="k">.</span><span class="o">.!</span>
0x0250: <span class="s2">6874 7470 3a2f 2f6d 6974 776f 726c 642e http://mitworld.</span>
0x0260: <span class="s2">6d69 742e 6564 752f 7669 6465 6f2f 3836 mit.edu/video/86</span>
0x0270: <span class="s2">37</span>00 0e6f 626a 6563 7445 6e63 6f64 696e <span class="s2">7</span>..objectEncodin
0x0280: 6700 c300 0000 0000 0000 0000 0009 0100 g...............
</pre></div>
<ul>
<li><span class="codehilite"><span class="k">string-object-marker</span></span> - A single byte of <code>02</code>.</li>
<li><span class="codehilite"><span class="o">unsigned 16-bit integer in big endian (network) byte order</span></span> - Specifies the length of the string to follow</li>
<li><span class="codehilite"><span class="s2">the actual string</span></span></li>
</ul>
<p>You can probably determine the proper string values by simply scanning the
ascii column, however I’ve highlighted specific bytes and strings in the output
so you can see how the <span class="caps"><span class="caps">AMF</span></span> protocol is structured.</p>
<p>From the above packet, we can gather the following parameters:</p>
<ul>
<li>app - <code>ondemand?_fcs_vhost=cp58255.edgefcs.net</code></li>
<li>swfUrl - <code>http://mitworld.mit.edu/.flash/player/Main.swf</code></li>
<li>tcUrl - <code>rtmp://80.154.118.22:443/ondemand?_fcs_vhost=cp58255.edgefcs.net</code></li>
<li>pageUrl - <code>http://mitworld.mit.edu/video/867</code></li>
</ul>
<h3 id="finding-play-parameters">Finding <em>play</em> parameters</h3>
<p>Now let’s do the same command as above, but for ‘play’ this time.</p>
<div class="codehilite"><pre><span class="nv">$ </span>tcpdump -X -r mitworld.cap | grep -B 5 -A 20 play
</pre></div>
<div class="codehilite"><pre>
0x0000: 4500 0093 f495 4000 4006 a321 c0a8 0228 E.....@.@..!...(
0x0010: 188f c74e 80ff 078f 931b 8661 1f34 958d ...N.......a.4..
0x0020: 8018 ffff 76b3 0000 0101 080a 0008 8921 ....v..........!
0x0030: 0ef1 08da 0800 101f 0000 5314 0100 0000 ..........S.....
0x0040: 0200 0470 6c61 7900 0000 0000 0000 0000 ...play.........
0x0050: 05<span class="k">02</span> <span class="o">0036</span> <span class="s2">616d 7073 666c 6173 682f 6d69</span> .<span class="k">.</span><span class="o">.6</span><span class="s2">ampsflash/mi</span>
0x0060: <span class="s2">7477 2d30 3132 3532 2d74 7261 6e73 706f tw-01252-transpo</span>
0x0070: <span class="s2">7274 2d6c 6f67 6769 6e67 2d73 6172 6d61 rt-logging-sarma</span>
0x0080: <span class="s2">2d33 306e 6f76 3230 3130</span> 0000 0000 0000 <span class="s2">-30nov2010</span>......
0x0090: 0000 00 ...
</pre></div>
<p>From this packet, which is sending the <em>play</em> command, we can extract our
<em>playpath</em>.</p>
<ul>
<li>playpath - <code>ampsflash/mitw-01252-transport-logging-sarma-30nov2010</code></li>
</ul>
<h2 id="testing-the-captured-rtmp-parameters">Testing the captured <span class="caps"><span class="caps">RTMP</span></span> parameters</h2>
<p>Now we’re going to use the rtmpdump tool to verify our connection parameters
are correct. We’re going to redirect the video to /dev/null since we don’t
actually want to download it.</p>
<div class="codehilite"><pre><span class="nv">$ </span>rtmpdump -r <span class="s2">"rtmp://cp58255.edgefcs.net/ondemand?_fcs_vhost=cp58255.edgefcs.net"</span> <span class="se">\</span>
--tcUrl <span class="s2">"rtmp://80.154.118.22:443/ondemand?_fcs_vhost=cp58255.edgefcs.net"</span> <span class="se">\</span>
--playpath <span class="s2">"ampsflash/mitw-01252-transport-logging-sarma-30nov2010"</span> > /dev/null
</pre></div>
<p>You can experiment here and try dropping off some of the parameters to see
which ones aren’t required. If the video connection works you will see a
progress meter. If you have an issues, you will see an error in the output.</p>
<h2 id="playing-an-rtmp-video-in-xbmc">Playing an <span class="caps"><span class="caps">RTMP</span></span> video in <span class="caps"><span class="caps">XBMC</span></span></h2>
<p>I found some helpful forum <a href="http://forum.xbmc.org/showthread.php?t=53156">posts</a> and a <a href="http://trac.xbmc.org/ticket/8971">trac ticket</a> with documentation on playing <span class="caps"><span class="caps">RTMP</span></span> streams.</p>
<div class="codehilite"><pre><span class="kn">import</span> <span class="nn">xbmc</span><span class="o">,</span> <span class="nn">xbmcgui</span>
<span class="n">app</span> <span class="o">=</span> <span class="s">'ondemand?_fcs_vhost=cp58255.edgefcs.net'</span>
<span class="n">swfurl</span> <span class="o">=</span> <span class="s">'http://mitworld.mit.edu/.flash/player/Main.swf'</span>
<span class="n">rtmpurl</span> <span class="o">=</span> <span class="s">'rtmp://80.154.118.22:443/ondemand?_fcs_vhost=cp58255.edgefcs.net'</span>
<span class="n">pageurl</span> <span class="o">=</span> <span class="s">'http://mitworld.mit.edu/video/867'</span>
<span class="n">playpath</span> <span class="o">=</span> <span class="s">'ampsflash/mitw-01252-transport-logging-sarma-30nov2010'</span>
<span class="n">li</span> <span class="o">=</span> <span class="n">xbmc</span><span class="o">.</span><span class="n">ListItem</span><span class="p">(</span><span class="s">'<span class="caps"><span class="caps">RTMP</span></span> Stream'</span><span class="p">)</span>
<span class="n">li</span><span class="o">.</span><span class="n">setProperty</span><span class="p">(</span><span class="s">'PlayPath'</span><span class="p">,</span> <span class="n">playpath</span><span class="p">)</span>
<span class="n">li</span><span class="o">.</span><span class="n">setProperty</span><span class="p">(</span><span class="s">'SWFPlayer'</span><span class="p">,</span> <span class="n">swfurl</span><span class="p">)</span>
<span class="n">li</span><span class="o">.</span><span class="n">setProperty</span><span class="p">(</span><span class="s">'PageURL'</span><span class="p">,</span> <span class="n">pageurl</span><span class="p">)</span>
<span class="n">xbmc</span><span class="o">.</span><span class="n">Player</span><span class="p">(</span><span class="n">xbmc</span><span class="o">.</span><span class="n">PLAYER_CORE_DVDPLAYER</span><span class="p">)</span><span class="o">.</span><span class="n">play</span><span class="p">(</span><span class="n">rtmpurl</span><span class="p">,</span> <span class="n">li</span><span class="p">)</span>
</pre></div>