Anyone familiar with wget and scripting??

[Hydro-Soil]

Hey All

I'm looking to archive subscribed threads and was wondering if anyone had ever scripted wget to do such a thing? If you still have the script, I would die for a copy ;D

Icmag uses a very popular forum structure and so do a lot of the other forums I visit. I'm looking for a script that would login as a user and make an off-line copy of all the 'subscribed' threads. This way they're still useful when the power or internet has issues.

Anyone??

 

[dreadvik]

I could write one tommorow for ya

 

[Hydro-Soil]

I could write one tommorow for ya

That would be totally wicked!



(Yes... I thought "The Incredibles" was a pretty cute movie. )

 

[dreadvik]

Okies gonna have a crack at it now Wish me luck! You will need your cookies file location and be able to run bash scripts so im guessin you got cygwyn or some ish

 

[h^2 O]

i use a wget editing thing for Ubuntu...kind of like "terminal" but it saves the script. IS this the same wget?

 

[dreadvik]

i use a wget editing thing for Ubuntu...kind of like "terminal" but it saves the script. IS this the same wget?

word not seen that! yeah it's the same ting I'm still struggling to find a cookies file thats usable so gonna try to export them from firefox

 

[dreadvik]

Ok a little long but your gonna have to grab https://addons.mozilla.org/en-US/firefox/addon/8154 I think IE6 allows you to export to a cookies.txt too though

 

[Hydro-Soil]

Ok a little long but your gonna have to grab https://addons.mozilla.org/en-US/firefox/addon/8154 I think IE6 allows you to export to a cookies.txt too though

Nice Extension. I can see how it would come in handy


Yes, bash is just fine as I have cygwin (or can install it again).

Sorry for no replies, been really busy lately. Much more active since I put together my first batch of cannapills

 

[dreadvik]

That's all good dood!

Cannapills? These sound like fun where do I get the knowhow?

Right the script was a menace to make cos vbulletin changed their authentification a while back and leave out one little secret and I don't have the source to work it out so it's still cookies only but exporting the cookie can be hard you may have to click remember me so the above plugin works I haven't tested this yet as it failed without for me. You also might be able to use another browser that will export the cookie or have a cookies.txt format you can point to directly. I hope this makes some sense

So here we go (rename attached copy from .txt to .sh and chmod u+x):

#!/bin/bash
# You need to edit cookies_file to point to your browsers cookie.txt if compatible with wget or export the cookie for the forum you wish to grab from and point to that.
# You need to call this script with the url to the forum you want to grab from including the forum directory.
#
# Commented lines below are for user/pass authentification instead of cookies but since a vbulletin security update it requires a salt from vbulletin db that is usally retrieved by some java script but I cant figure out how to get at. If you can it needs to be added to the end or beggining of the $pass in the md5 line and the comments removed.
#
usage="$0 <forum url>"
if [ "$1" = "" ]
then
echo $usage
exit
fi
url="$1/subscription.php"
cookies_file="$HOME/Documents/cookies.txt"
echo "Using cookies file: $cookies_file (Edit this script to change)"
#read -p "Enter Username: " user
#read -sp "Enter Password: " pass
#echo "Generating md5 password hashes"
#vb_login_md5password_utf=`echo $pass | md5`
#vb_login_md5password="$vb_login_md5password_utf"
#echo "Logging into $url as $user"
wget -q --load-cookies $cookies_file --save-cookies=$cookies_file --keep-session-cookies -O /dev/null $1
#wget -q --load-cookies $cookies_file --save-cookies=$cookies_file --keep-session-cookies --post-data="vb_login_username=$user&vb_login_password=&s=&securitytoken=guest&do=login&vb_login_md5password=$vb_login_md5password&vb_login_md5password_utf=$vb_login_md5password_utf" $1/login.php?do=login
echo "Getting subscribed posts list from $url"
wget -q --load-cookies $cookies_file --save-cookies=$cookies_file --keep-session-cookies $url
echo "Saving posts to $PWD/posts directory"
mkdir -p posts
cd posts
for i in `cat ../subscription.php | grep 'showthread.php?goto=newpost' | awk -F'"' '{ print $2 }'`
do
wget -q -E -H -k -K -r -l1 -p $1/$i
done
echo "Cleaning up"
cd ..
#rm login.php?do=login
rm subscription.php

 

[dreadvik]

Here's an example:

./subscribed_posts_scrape.sh http://icmag.com/ic

Then point your browser at the posts directory in the current directory as the output suggests.

 

[Hydro-Soil]

You rock Dreadvik!

I'll get feedback for you as soon as I can.

As for the pills... Follow the BHO guides here to make clean Butane Honey Oil, heat it in a double-boiler with olive oil or another compatible oil and then soak it up with an edible powder.

The last run I did I used the frosty trim and then powdered the spent trim to soak up the BHO and Olive oil. You can also grind the bud/leaf itself to a powder and mix it directly with the olive oil in the double-boiler.

Use VERY little olive oil.... it takes a LOT of powder to make it manageable to where it won't melt the gel capsule while waiting to be eaten in the freezer.

Edit: Oh yeah.... be careful since the effects last longer than smoking and eating too much can be..... unpleasant.

 

[dreadvik]

No probs dood

Man these pills sound good and I already read the BHO so I'm good to go once I decide on what powder to use I wonder if it's easy to do that candy shell coating thing

 

[Hydro-Soil]

No probs dood

Man these pills sound good and I already read the BHO so I'm good to go once I decide on what powder to use I wonder if it's easy to do that candy shell coating thing

Once it's the oil is absorbed into the powder.... I just use empty gelatin capsules you can get in the health food stores or online.

 

[McSnappler]

They sound easy to mail, too!

 

[10k]

tag'age

 

[ParanoidFarmer]

dreakvik, we meet again. Seems our tastes are similar.
I about an hour today writing a reverse APR tool that we already had in our network engineers toolkit. Ha. Good thing i didn't start on the threaded version. :L

 

[Hydro-Soil]

They sound easy to mail, too!

Well.... if you use olive oil and not enough powder, they don't stay 'good' at room temp very long.

Possibly with a dryer mix and tougher capsules? It would last long enough.... I'm still working on em

CannaBrex makes a kit with extra goodies but I'm not interested in the extras just yet. The pill kit, spatula and stuff is what I want. Love the jelly jar reagent chamber or whatever.

 

[dreadvik]

This is making me wish I bought the capping kit for my nootropics id be good to go already then still dont think it cost much $10 and then a bit for a bag of gel caps

 

[dreadvik]

dreakvik, we meet again. Seems our tastes are similar.
I about an hour today writing a reverse APR tool that we already had in our network engineers toolkit. Ha. Good thing i didn't start on the threaded version. :L

Hehe yeah both in the game I'm guessing APR heh ya lost me there i'm not quite a programmer Unless thats a typo for ARP then I'm your man Just don't get me started on STP

 

[ParanoidFarmer]

Yeah i meant ARP, sorry. I have a bad habit of not proofreading what i write very well.