OpenSci.Info Blog

Public Library of Science (PLoS) Automated Download Methods

2009-02-11T15:06:00.001-08:00

Public Library of Science Automated Download Methods

Via email from PLoS Webmaster

(highlights & formatting added)

To retrieve an entire volume, we recommend using the PubMed Central FTP service. I’ve included other options below as well-- a PHP Script for downloading XML files and using wget to get PDF and XML files (just change the HTTP address to the location of the PLoS Biology articles).

PubMed Central FTP Service

The PMC FTP Service may be used to download the source files for any article in the PMC Open Access Subset, associate PMC articles with identifiers such as: PubMed IDs, DOIs, Manuscript IDs, ISSN, etc., and can used as a source for data mining.

For more information: http://www.pubmedcentral.nih.gov/about/ftp.html

PMC also has a single archive file (~1Gb) that contains XML (and only XML) files for ALL PMC open access articles. This was created for users who need PMC XML for data mining and processing purposes, but do not need PDFs, images, or supplementary data.

For more information: http://www.pubmedcentral.nih.gov/about/ftp.html#XML_for_Data_Mining

Using a PHP Script

The XML and PDF files are available for each article from the PLoS journal websites. You’ll need to create a script to download these files. Fred Howell wrote a script (http://www.neurogems.org/fetchplos/) to download XML files. This script can be easily modified to also download the relevant PDF files.

PLoS articles are all tagged to parse against the NLM/NIH journal publishing DTD. The latest version is DTD 2.2

Here’s the link to the Preview XSL Transform that the NLM/NIH group has written for their DTD: http://dtd.nlm.nih.gov/tools/

And you find the DTD in the DOCTYPE tag of the XML article, or
 
< !DOCTYPE article PUBLIC "-//NLM//DTD Journal Publishing DTD v2.0 20040830//EN" "http://dtd.nlm.nih.gov/publishing/2.0/journalpublishing.dtd" >
 
All articles can be redistributed and reused according to the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.5/).

Using wget

Here’s an example script to use wget to fetch PDF files from PLoS ONE articles 0000001 - 0000009:

#!/bin/sh

for i in `seq 1 9`

do

    echo "Fetching journal.pone.000000$i.pdf..."

    wget -q "http://www.plosone.org/article/fetchObjectAttachment.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.00

0000$i&representation=PDF"

    echo journal.pone.000000$i.pdf >> filelist

    # don't bash the servers!

    sleep 10

done

Here’s an example script to use wget to fetch XML files from PLoS ONE articles 0000001 - 0000009:

#!/bin/sh

for i in `seq 1 9`

do

    echo "Fetching journal.pone.000000$i.pdf..."

    wget -q "http://www.plosone.org/article/fetchObjectAttachment.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.00

0000$i&representation=XML"

    echo journal.pone.000000$i.xml >> filelist

    # don't bash the servers!

    sleep 10

done

Dynamic FeedFlare using AppJet server side Javascript

2008-10-15T02:28:00.001-07:00

Dynamic FeedFlares using AppJet server side Javascript

In this example, if the item category is "Music" then return a custom link, else show nothing.

AppJet

http://appjet.com/app/3454754/source

/* appjet:version 0.1 */

page.setMode("plain")

print(html("""<FeedFlare>
"""));

if (request.params.category) {

    if (request.params.url) {
    
        if (request.params.category=="Music") {
    
            print(html("""<Text>Stream with BitLet</Text>
            <Link href="http://www.bitlet.org/music/play?torrent="""
            +request.params.url
            +"""" />
            """));
        }
    }
}


print(html("""</FeedFlare>"""));

FeedFlare

http://pastebin.ca/raw/1227586

<FeedFlareUnit>
<Catalog>
<Title>BitLet Stream Music</Title>
<Description>Stream using BitLet.org if category is Music</Description>
</Catalog>
<DynamicFlare href="http://bitletfeedflarestreamif.appjet.net/?category=${a:category[1]/@term}&url=${a:link[(@rel='enclosure')]/@href}"/>
</FeedFlareUnit>

FeedBurner and FeedFlare enclosure custom links

2008-10-14T05:11:00.000-07:00

Feed with embedded custom links
1. FeedBurner
2. FeedFlare
Feed with BitTorrent enclosures
Add and configure feed
1. SmartCast
2. SmartFeed
FeedFlare

Feed with embedded custom links

FeedBurner

Allows feeds to be displayed in HTML or compatible format.
FeedFlare
Adds custom links based on feed item information.

Learn more about the FeedFlare API.

FeedFlare testing scratchpad.

Feed with BitTorrent enclosures

Original

http://friendfeed.com/rooms/opensciinfo-bittorrent?format=atom
Output

http://feeds.feedburner.com/opensciinfo/bittorrent
Example display

http://wiki.opensci.info/bittorrent

Add and configure feed

Enable from optimize menu

SmartCast

Increases compatibility with various podcast or multimedia enclosures. Use default options.

SmartFeed

Some feed readers prefer certain versions of RSS and Atom.

FeedFlare

An XML snippet that uses feed item values to create custom links in the description.

FeedFlares can be added directly from a URL and applied to any feed.

BitLet

BitLet Download

Browser based Java applet to download torrent files

http://pastebin.ca/raw/1226774

<FeedFlareUnit>
<Catalog>
<Title>BitLet Download</Title>
<Description>A link to download enclosure BitTorrent file using BitLet.org</Description>
</Catalog>
<FeedFlare>
<Text>BitLet Download</Text>

<Link href="http://www.bitlet.org/download?torrent=${a:link[(@rel='enclosure')]/@href}"/>
</FeedFlare>
</FeedFlareUnit>
BitLet stream

BitLet Stream

Browser based Java applet to stream mp3 and ogg music files

http://pastebin.ca/raw/1226829

<FeedFlareUnit>
<Catalog>
<Title>BitLet Stream</Title>
<Description>Stream music from BitTorrent file using BitLet.org</Description>
</Catalog>
<FeedFlare>
<Text>BitLet Stream</Text>
<Link href="http://www.bitlet.org/music/play?torrent=${a:link[(@rel='enclosure')]/@href}"/>
</FeedFlare>
</FeedFlareUnit>
BitLet stream dynamic

More about dynamic FeedFlare

BitLet Stream
Browser based Java applet to stream mp3 and ogg music files, link appears for posts in the category "Music

http://pastebin.ca/raw/1227586

<FeedFlareUnit>
<Catalog>
<Title>BitLet Stream Music</Title>
<Description>Stream using BitLet.org if category is Music</Description>
</Catalog>
<DynamicFlare href="http://bitletfeedflarestreamif.appjet.net/?category=${a:category[1]/@term}&url=${a:link[(@rel='enclosure')]/@href}"/>
</FeedFlareUnit>
TorrentRelay

TorrentRelay Download
Server processes the torrent and lets the user download the files like regular web downloads

http://pastebin.ca/raw/1226765

<FeedFlareUnit>
<Catalog>
<Title>TorrentRelay Download</Title>
<Description>A link to download the BitTorrent enclosure files using TorrentRelay.com</Description>
</Catalog>
<FeedFlare>
<Text>TorrentRelay Download</Text>
<Link href="http://www3.torrelay.com/fresh/online.pl?url=${a:link[(@rel='enclosure')]/@href}" />
</FeedFlare>
</FeedFlareUnit>

TorrentRelay mobile

TorrentRelay Mobile Download
Provides a slimmer web page format for downloading the torrent on mobile device browsers

http://pastebin.ca/raw/1226785

<FeedFlareUnit>
<Catalog>
<Title>TorrentRelay Mobile Download</Title>
<Description>A link to download an enclosure BitTorrent file using TorrentRelay.com mobile download.</Description>
</Catalog>
<FeedFlare>
<Text>TorrentRelay Mobile Download</Text>
<Link href="http://www3.torrelay.com/fresh/online.pl?url=${a:link[(@rel='enclosure')]/@href}&amp;ajax_uploader=yes"/>
</FeedFlare>
</FeedFlareUnit>

ImageShack
ImageShack Download
Provides a web page to download the torrent files, requires free registration.

http://pastebin.ca/raw/1227565

<FeedFlareUnit>
<Catalog>
<Title>ImageShack Download</Title>
<Description>A link to download an enclosure BitTorrent file using ImageShack.com torrent download.</Description>
</Catalog>
<FeedFlare>
<Text>ImageShack Download</Text>
<Link href="http://tor.imageshack.us/tor/?andthen=1&url=${a:link[(@rel='enclosure')]/@href}&submit=GO&action=geturl"/>
</FeedFlare>
</FeedFlareUnit>

Adapting Elearning Twitter Feeds to OPML

2008-09-30T14:27:00.000-07:00

Centre for Learning & Performance Technologies compiled a list of active Twitter users in the field of education eLearning Tech blog asked if there were a way to add Twitter follows similar to a news feed OPML OUseful.Info was able to make a Yahoo Pipe which aggregated the Twitters into an RSS feed I tried to find a way to adapt the original list to OPML format Grab the OPML (updated 9/30) or view it as a blogroll. Check out the Google spreadsheet used to generate the OPML source.

2008-07-16T09:57:00.000-07:00

Python image stuff comparison

Functions

Nodebox script tufte_nodebox_forcepush.py

NodeBox/QT

requires QT

Shoebox

requires: Cairo 1.4.2

Python script comparisonchartgenerator.py

Python Imaging Library (PIL)
Tkinter

System platform & hosting

Shared hosting

Debian (sarge) Linux

Debian (etch) Linux

cairo 1.2.4-4.1

Google App Engine

Utility Mill

Dedicated server / VPS / local

PIL, Tkinter

comparisonchartgenerator.py

Cairo 1.6.4

Shoebox

QT NodeBox

Example Implementations

Dedicated server / VPS / local

Debian (Lenny or Sid) (sidux also tested?)

NodeBox/QT Installation

sudo apt-get install subversion
sudo apt-get install pyqt4-dev-tools
svn co http://dev.nodebox.net/svn/nodebox/branches/try-qt/
cd try-qt
python qtrun.py

wget http://media.juiceanalytics.com/downloads/tufte_nodebox_forcepush.py

ShoeBox Installation

Install Cairo
sudo apt-get install libcairo2-dev
Install PyCairo
sudo apt-get install python-cairo-dev
Download and install ShoeBox
wget http://shoebox.sollec.org/files/shoebox-0.2.tar.gz

tar -xzvf shoebox-0.2.tar.gz

cd shoebox-0.2

sudo python setup.py install
Run ShoeBox with tufte_nodebox_forcepush.py

wget http://media.juiceanalytics.com/downloads/tufte_nodebox_forcepush.py

Shoebox install on sidux
-install cairo

Another example - VirtualBox
Download VDI by BT
Sidux 2008-02
http://www.mininova.org/tor/1579459

Install QTNodeBox
-

OpenSci.Info Blog

Public Library of Science (PLoS) Automated Download Methods

To retrieve an entire volume, we recommend using the PubMed Central FTP service. I’ve included other options below as well-- a PHP Script for downloading XML files and using wget to get PDF and XML files (just change the HTTP address to the location of the PLoS Biology articles).

PubMed Central FTP Service

Using a PHP Script

Using wget

Dynamic FeedFlare using AppJet server side Javascript

Dynamic FeedFlares using AppJet server side Javascript

AppJet

FeedFlare

FeedBurner and FeedFlare enclosure custom links

Feed with embedded custom links

FeedBurner

FeedFlare

Feed with BitTorrent enclosures

Original

Output

Example display

Add and configure feed

SmartCast

SmartFeed

FeedFlare

BitLet

BitLet stream

BitLet stream dynamic

TorrentRelay

TorrentRelay mobile

ImageShack

Adapting Elearning Twitter Feeds to OPML

Python image stuff comparison

Functions

System platform & hosting

Shared hosting

Google App Engine

Utility Mill

Dedicated server / VPS / local

Example Implementations

Dedicated server / VPS / local