9 votes

Linux Users: A Script to Download Your Posts

A while back, I recall user BILL3 asking about a way to download the content he had created in case he may want to compile it later into a larger project such as a book. For Linux users, here is a script that will do just that.

######################################################################

#!/bin/bash

# Usage: DP-post-downloader.sh "url of myposts page"
#
# For instance, DP-post-downloader.sh http://www.dailypaul.com/myposts/56701
# would download the first page of posts from user dwalters to the current directory

url=$1

w3m -dump_source $url > temp.file
start=`cat temp.file | grep -in "tbody>" | sed -r 's~(.*):(.*)~\1~g' | awk '{if (NR == 1) {print $0}}'`
end=`cat temp.file | grep -in "tbody>" | sed -r 's~(.*):(.*)~\1~g' | awk '{if (NR == 2) {print $0}}'`
list=`sed -n "$start,$end"p temp.file | grep href | sed -r 's~(.*)"(.*)"(.*)~\2~g'`

for i in $list; do
wget "www.dailypaul.com"$i
done

rm temp.file

######################################################################

Of course, you can name the script whatever you wish. It doesn't have to be DP-post-downloader.sh.

Instructions (if you're not familiar with using scripts):

From the command line,

1. Type w3m and press [Enter]. If the command is not found, type sudo apt-get install w3m and press [Enter]. This will download the command line browser w3m.

2. Change directories to where you wish to download your posts. I will use /home/$USER/Documents/WRITING/DAILY_PAUL

mkdir /home/$USER/Documents/WRITING/DAILY_PAUL

cd /home/$USER/Documents/WRITING/DAILY_PAUL

3. Create the script file. This will open a blank file in the Vim editor (a not-so-primitive but primitive looking text editor).

vi DP-post-downloader.sh

4. Copy the script text. Vim doesn't support any of that right clicking garbage. Instead, simply highlight the script (starting at #!/bin/bash) that is at the beginning of this post. Once highlighted, go to your Vim window and press the key [i] (lowercase i stands for "insert"). Now, press the center mouse button. Whamo! It is now pasted. Press the [Esc] key, and then, type :wq (meaning write and quit) and press [Enter].

5. Change the permissions for the script. At this point, you should be back at the command line. So, type:

chmod 700 DP-post-downloader.sh

6. Download your posts! Get the url of your "myposts" page by going to your profile and clicking on the "All" link beside "Recent Posts" and copying the url from the address bar. Like in Vim, you can paste something you've copied to the command line by pressing the center mouse button. The command should look something like this (but with your own url):

./DP-post-downloader http://www.dailypaul.com/myposts/56701

7. Repeat step 6 for any additional "mypost" pages you may have.

Enjoy!




Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

I updated the instructions, I forgot to write to the file before

exiting vim.

Fixed now.

ChristianAnarchist's picture

Great! I've wanted a way to

Great! I've wanted a way to download all my BS I've posted over the years. Too bad I don't have any Linux machines and don't know how to run the script if I did. Anyone want to offer to grab my posts and send them to me in a zip file??

Done

Shoot me an email at dwalters@uark.edu

(Please don't everyone start lining up asking me to do the same! It could quickly turn into a nightmare!)

Joη's picture

here we were, working on that functionality for regular users

and you had to go write a script solution for it! Well, good.

"You underestimate the character of man." | "So be off now, and set about it." | Up for a game?

I had 290 posts according to wc -l

I sure wasn't going to copy and paste every link!

If another tbody element was added to the "myposts" page before the current one that is there, the script would fail (or download the links associated with the new table).

Joη's picture

your profile count says:

Nodes : Comments
302 : 2693
(as of comment timestamp)

"You underestimate the character of man." | "So be off now, and set about it." | Up for a game?

Perhaps a fraction of them were unpublished...

Not sure. I haven't checked. Do those still show up on a users list?

Joη's picture

yep

.

"You underestimate the character of man." | "So be off now, and set about it." | Up for a game?

Sure enough. 403 when w3m hits one.

...

I've now included instructions.

With a little time on the command line, these things can become second nature.