A big problem with the upgrade was trying to switch from a SQLite to MySQL database. To do this, I had to export all my entries (easy) and create an empty MySQL database (also easy). Then I need to reconfigure movable type to look at the MySQL database, instead of the SQLite databse (super easy). Then I rebuild my blog, delete the old SQLite database, and have a Guinness to celebrate (pending.)
Still with me?
Okay. So here’s the problem: my export file is 37M, and the server tiems out when it’s trying to import that file. I didn’t know what to do about that, and all this week, I haven’t had the time to figure out a solution. Luckily, Jay told me that I can break up my thirty-seven megabyte text file into as many smaller text files as I want, and import each one. They won’t overwrite each other (which is what I thought they’d do), and when I’m done, I should have a fully functional weblog.
However, thirty-seven megs of plain text is a ton of data, and I don’t have time to cut it all up, upload, it, and repeat that process probably ten times or so. I probably won’t have time until next week. Luckily, I have 26 days remaining before Typepad comes around with a baseball bat and a paper bag.
A quick thought on TypePad: It’s awesome. The interface I’ve been using to update is the nicest WYSIWYG interface I’ve ever used for anything, and the way they let you choose templates and styles is awesome. In fact, if TypePad had existed back when I started WWdN, I doubt I would have ever configured and installed my software on my own server.
When WWdN comes back, expect it to look and feel a lot like this, but without any of the fixed-width stuff.
Discover more from WIL WHEATON dot NET
Subscribe to get the latest posts sent to your email.
I just want to say. That picture rocks.
I love using WP, and sorry you’re having troubles with your site. I came here with an agenda though. I know you love boobies, I love boobies too—I have some myself. But this comment really is about Breast Cancer and I know that you and your wife support that cause. So I’d like to invite you to come over to http://www.boobiethon.com/
I’d like to invite your readers too. It’s a great cause, and you get to see those beautiful boobies while you’re at it.. Thank you Wil for all that you and Anne have done for this cause.
You might explore the split command from an OS X or Linux prompt. That would split your huge file into manageable chunks. You’d probably need to copy and paste a handful of header commands into the top of each output file so that the new SQL database will know where to store the data.
Wil,
I’m oing to second Michael’s comment about the split command. It’s a standard unix command that breaks files into chunks. Since you’re breaking up an SQL export, you should split by lines rather than size.
I have to do this all the time on multi-gigabyte log files. Works like a charm.
Since you’ve already rebuilt the database, as long as your sql export has all the right insert statements, it should accept being cut up without any problem.
Like I said though, use line numbers, i.e. split -100000 [filename]. If you try it by size it’ll break off in the middle of something probably important.
Tim beat me to it.
I was going to say the same thing. If you cut it up by size it might split up in the middle of an INSERT statement and that wouldn’t be good.
Don’t bother splitting the text file. Just change the maximum allowable packet size variable to more than 1 meg in your server settings. Muuuuucccchhh easier.
Regarding the split command, there’s a nice tutorial here:
http://www.macmove.com/mm/2005/09/29/40gb-sure-seemed-like-a-lot-a-tutorial/
Or, if you’re not a Terminal guy, this presents a great way to learn to use Automator.
I LOVE Typepad. 🙂
Ditto on the TypePad love. I like MovableType because it lets me tinker and play and configure everything to my heart’s content, but TypePad just makes everything so elegant and easy.
Hey Wil,
I actually suggest taking a PHP approach to your issue. Write a script that can parse out the file and run about 1000 lines at a time and when that’s done it runs the next 1000. It’s really easy to do. Email me if you would like me to send you an example script you can just plug your own information into.
To WWdN:
Here’s some virtual chicken soup…get well soon!
Gilder
San Antonio
Hey Wil,
I’ve got a Typepad account as well and discovered that you can defeat the horrid “fixed width” syndrome by revisiting your design settings and changing your main column width to “flexible” or some derivation thereof. Seems to work for me. Like setting your main column to 100% to fill the screen that it’s being viewed on. Give it a go.
WWdN, get well soon!
Glad to hear you’ve found a solution and I’m sorry to hear you’re having trouble. Best of luck figuring it out!
And good luck with that Guinness. *LoL* That stuff has a taste you really have to get used to… well, at least I do. But it’s good, no doubt. I think you’re required to drink it if you’re Irish. *LoL*
Along the lines of boobiethon, when I saw this I thought of you:
http://www.starnow.com/Models-wanted/poker_models_wanted.asp
It’s got breast cancer charity, poker, hot models, exotic locations…
Typepad rules! It allows you enough room to tinker/modify, w/out requiring you to do coding.
On the suggestion above to use the ‘flexible’ setting: it applies to the center section but has one problem. When you use it, that space can shrink down to an inch across for those with 14-inch monitors. Makes for a *very* long front page.
Also Typepad, and SixApart in general, are about to release a whole new swath of upgrades and features to their weblog software across the board. I’m a longtime Typepad user and really looking forward to that.
Wil,
My name is Shannon and I am a fan of your blog and writing. I am from Long Beach, Mississippi where we were hit hard by Hurricane Katrina. Long Beach is a small town in between Gulfport and Pass Christian. Unfortunately I lost both my home (it was 100 feet from the beach and is now what my husband and I lovingly call “our slab”) and the local library I worked as a reference librarian in. I am writing to ask if you would be willing to donate your books to our library. I would really love for your books to be some of the first ones we put on our shelves. We just finished building a new children’s library before the storm and our big building and children’s library have now been condemned and will have to be demolished. I think books are very important to our community to help keep their minds off the horror we see everyday as we drive down our streets. We would really appreciate the donation. I purchased and read your book before the storm and planned to donate my copy, but unfortunately it was lost. If you would like more information on the library e-mail me at [email protected].
I tried sending this to your e-mail, but it wouldn’t work. Sorry for posting this on your comments!
Thanks so much!
Shannon
Wil,
I’ve used a utility called BigDump (yes, bad name) before with good results. It’s designed to get around this exact problem, and automatically handles everything for you – I’ve used it before with large MySQL databases and it’s saved me a bunch of time.
http://www.ozerov.de/bigdump.php
Hope this helps!
Don’t rush, go take in the movie “Serenity”..it kicks butt.
hey will, just an idea but why dont you put your temporary blog up in buzznet, I know wendy may has jumped ship to using this over blogspot. just an idea… (oh and the text file should have some delimiter that you can find using the “find” option to make it easier to identify where you should split. Put cursor mid way down txt file, then press find for the delimiter to split it in to) its a far easier way to do it than what ive seen above. only problem is to how to find out what the delimiter is.
Noel
Wil – I work for a hosting company and we have a user who has a script *exactly* for this purpose.
http://forum.powweb.com/showthread.php?s=&threadid=20022
You can upload via FTP and then populate to a mysql database just by filling out the form using your full root path. (change the extension to php, but then you already knew that. 😉 )
Good luck on the site, I’ve my own 10 or so that go through the annual renewal/update.
btw- in case you haven’t heard “A Clear Horizon” just landed in amazon.co.jp last month. Making it available in US, UK, Italy, and now Japan. The book has done fairly well, thanks to the writers of thesoapbox and the space you gave us to get together. 🙂
=)
Jade
Well, I hope WWdN comes back up to speed soon. In the interim, I would like to announce that the winner of the Wil Wheaton Memorial Poker Tournament at Linucon 2 this past weekend was James Ernest of Cheapass Games.
He avenged Wil’s honor by defeating Wil Upchurch, winner of last year’s tourney (over our own beloved Wil Wheaton at the final table).
–Chase Hoffman
I should mention that I got 3rd at the Wil Wheaton Memorial Poker Tournament just like last year. Thanks Wil, due to that tournament last year I now play on average 3-5 times a week at freeroll tournaments around town. This year I lost on skill, not just on luck.
I have another idea that might save you a bunch of time. I’m assuming that you have a set of mySQL commands in your 37MB file and that you are trying to import this massive file via phpMyAdmin or something like that.
If you have either SSH of FTP access, it should be much easier to upload the file to your server. Once it’s on your server you should be able to use SSH to log in and import the file via the mySQL command line interface OR you should be able to point phpMyAdmin to use a file residing on the local server. Hope that helps and let me know if you need any help.
Hey, Wil, have you given up on resurrecting your old blog, or do you still need/want help here? I’d be glad to throw a script together to help you get things back in order, if you’d like. Just drop me an email and we’ll work out the details.
(If my overzealous spam filter bounces your mail as spam, try sending the message as plain text intead of HTML.)