Amazon Cloud, iTunes, and Perl

It was not long ago that I purchased a Samsung Admire as my first smartphone.  I bought it for two reasons: 1) It runs on Linux 2) It multi-tasks.  With the Android OS, I believed it was possible to ditch my GPS and my iPod while out on runs, rides, or for any other reason.

The Samsung has done well in both respects, but this is not a review of the Samsung nor of the Android OS. Instead, when I chose to use the Android as my ‘music player of choice’, I fell into a serious conflict problem…and it’s name is Amazon vs. iTunes.

I know of no iTunes app for the Android OS.  In addition, I have a rather large collection of music already saved in a format for iTunes.  However, I do have access to the individual music files because I was only running iTunes in a virtual XP box on my Ubuntu laptop.  I thought that porting them over to an mp3 format and uploading to my free cloud drive on Amazon would be a breeze.

No way.  Not happening. I could use gzip and ffmpeg to extract and convert the m4a files in my iTunes folder.  The converted mp3 files also played fine and could be uploaded to the Amazon Cloud Drive.  But, that is where happiness ended.

When I used the Amazon Cloud Player app on my Android OS phone, the files I had uploaded all got dumped into the ‘Unknown album’ folder.  Most of the song names also ended up with a great deal of gobbledy-gook added to the song names and made it difficult to read.  However, any songs I purchased on Amazon appeared just perfectly peachy in the Cloud Player as well as in the music player of the Samsung.  OK, re-purchasing all my already purchased songs on Amazon was not an option.

I did no short amount of searching and looking for a solution for the problem when I happened to accidentally stumble upon the answer.  It seems that Amazon’s Cloud Player does not care what the filename of the song is.  It only cares what is kept in the “metadata tags” of the song.  Metadata tags are text variables (my interpretation) that give descriptive information about the audio, video, or image file to which they belong.  With an experiment, I discovered that when I ran ffmpeg to convert from the m4a to an mp3 file, metadata tags were not copied as well. After conversion, I ran a converted mp3 through mplayer and saw that only the song name was there; no artist, no album.  Additionally, even if they had been, they would have been incorrect as m4a utilizes the ‘artist’ tag while the mp3 uses ‘author’ for the same variable.

Now, I did find a way to do the conversion and load the metadata tags all at once.  However, that left me with the prospect of copying the m4a to a temp folder, extracting the m4a from the gzipped file, extracting the metadata tags, and then building a long command line for each of over 500 music files.  This did not count also that I wished to strip out leading numerals, punctuation, and replace the spaces with underscores at the same time.  I needed a script.

Since I have been trying to learn Perl, I thought this was a good time to try it out.  It seemed easy…right?  While it was not the easiest I have done, it only took about four hours total work on the scripts over two days.  I wrote two scripts: one to correctly rename the music files how I wanted them, and one to create the conversion process.  In between these, I would use gunzip to extract the compressed m4a files.  If you are not concerned about oddly named files, you may only need the conversion script.

I did this on an Ubuntu laptop.  I do know that one can run ffmpeg and Perl on a Windows machine.  However, I do not know what changes might be required to accomplish this task on that platform.  You will need another helper app also; exiftool.  I don’t know if this is available on Windows or not.  If you can find another command line tool that can extract metadata tags; you should be able to recreate this on Windows.  I did try to use ffmpeg to extract tags, but it’s output does not correctly go to STDOUT, nor does it seem to go to STDERR.

Also, though I am publishing both of these scripts for anyone to use or modify, I ask for credit at least on the m4aconvert.pl if you do modify and re-publish.  I am sure that both of these can be gussied up if someone were to take the time.  The itunesren.pl script is based on answer given on Perl Monks forum by John Lawrence.  All that I added is the extra substitution patterns I needed for my files.  The bulk of that script is Mr. Lawrence’s.

Both of these scripts will work better if you add them into your local BIN folder which should be on your PATH.  Look up how to do that if you need to.  Also, both work best if the only files in the folder you are working on are the gzipped m4a files.  The presence of other files will cause both to emit errors or malfunction entirely.

First step; ITUNESREN.PL

Usage: itunesren.pl *

#!/usr/bin/perl
#
# Renaming music files containing
# spaces, punctuation, and leading
# numbers.
#
# modified by DEV on 5/24/2012

use strict;
use warnings;

foreach $_ (@ARGV) {
my $oldfile = $_;         #copy input record into oldfile
s/[0- 9] //g;                               #clean out numerals
s/\.ma\.gz/\.m4a\.gz/g;       #put 4 back in m4a
s/^ //g;                                        #remove leading space
s/& //g;                                       #remove ampersands
s/[()]//g;                                    #remove parentheses
s/-//g;                                         #remove dashes
s/ /_/g;                                       #replace spaces with underscore
rename($oldfile, $_);             #now, rename the file $_ into $oldfile

}

Some notes you will want to know are: the ‘s/ /_/g; line has an actual SPACE between the first two slashes, and the ‘s/-//g;’ substitution was not tested as I had used another substitution that removed the dashes (and most everything else as well).  It might have to be ‘escaped’ instead (i.e. s/\-//g;).

If you notice that most of the lines in this script do not seem to be acting upon any variables, that is because Perl has a built in ‘input record’ variable ($_).  Each of my substitutions is working upon this variable and so it does not have to be actually mentioned.  I store the original filename at the beginning and then modify the input record during the script before loading it back into the real filename at the end.

Second Step: You now should have all your filenames like you like them (or how I like them).  It’s possible not to use step one if you don’t wish to.  At this point all you need is the command ‘gunzip *’) to extract the compressed files.  You’ll be left with must the m4a native files.

Third Step: m4aconvert.pl

Usage: ls|m4aconvert.pl

#!/usr/bin/perl -w
#
# Convert itunes *.m4a files to *.mp3 files
# Will read entire directory contents as input
# Also reads m4a metadata and writes to new mp3 file
# Requires: ffmpeg and libmp3lame codec
# Written by: Dave Vest on 5/26/2012
#
# Usage: ls|m4aconvert.pl

# Loop through all input files
while (<>){

# Chomp is used liberally to remove \n from input
# Required in order to build final ffmpeg stream.
# exiftool extracts metadata tag according to value
# You supply the tagname and it extracts the value

chomp($tag1 = `exiftool -q -title $_ `);

# Build metadata tag variable from exiftool return
# exiftool returns; “Title : title”
# split at ‘:’ and return what is to its right

$title = ” -metadata title=\”" . (split /: /, $tag1 )[-1] . “\”";

# Do it again for each tag.

chomp($tag1 = `exiftool -q -artist $_ `);
$author = ” -metadata author=\”" . (split /: /, $tag1 )[-1] . “\”";
chomp($tag1 = `exiftool -q -album $_ `);
$album = ” -metadata album=\”" . (split /: /, $tag1 )[-1] . “\”";
chomp($tag1 = `exiftool -q -year $_ `);
$year = ” -metadata year=\”" . (split /: /, $tag1 )[-1] . “\”";
chomp($tag1 = `exiftool -q -genre $_ `);
$genre = ” -metadata year=\”" . (split /: /, $tag1 )[-1] . “\”";

# chomp removes ‘\n’ from input filename
# get input filename into a variable.

chomp;
$infile = $_;

# split the .m4a off of the filename

$fname = (split /\./, $_ )[0];

# Put .mp3 onto filename for output file of ffmpeg
# also add a space in front for the command line

$outfile = ” ” . $fname . “\.mp3\n”;

# Now, build the command line string for ffmpeg
# for the current filename and set up all
# metadata tags. Note: ‘author’ is ‘artist’ for mp3 files.

$ffmpegstr = “ffmpeg -i ” . $infile . ” /
/ -acodec libmp3lame -ab 160k “
/ . $title . $author . $album
/ . $year . $genre . $outfile;

# Call system to run ffmpeg for the current file.

system($ffmpegstr);
}

Some important notes for this file are: The line continuations that I have used (/) in the line where the command is built should not be in your final script.  My script line was too long for WordPress and needed shortening.  Remember to remove these when you create your script, and the chomp’s are critically important.  Everything that you read from exiftool or from the input record contains a newline character (\n) that must be stripped before it can be utilized in the command line.  I spent several hours trying to figure out why ffmpeg kept telling me I had no output file.  It was because I read in the $infile variable and it came with it’s own newline.  That truncated my command and caused the rest of the variable to end up on the second line and thus, no output file name.  Finally, those quote-looking things around the exiftool commands are back-quotes.  They are not regular quotes.  Back-quotes work like the system() function in the last line, but return the output value of the system command.

Both of these scripts should work as is, but I recommend segregating copies of your iTunes song files into a temp folder of some kind.  DO NOT TAKE ACTION ON YOUR ORIGINAL FILES!  These scripts are destructive.

When complete, you will be left with a folder of m4a and mp3 files with the same names.  Do a wildcard delete (rm *.m4a) of the original iTunes files and you will now be ready to upload your mp3 files to the Amazon Cloud (or Cloud Player).  The Cloud Player will interrogate the metadata tags and properly set up your song, album, and artist names so that things look correct.  This will also allow the Cloud Player app on your Android phone to hunt down the album art.  All except the album art will carry through as you download songs to your Android phone.

There are lots of possibilities from here.  There could be a change to just allow you to specify the folder to take action upon as a command line argument.  Someone could gussy it up for a GUI environment.  It could also use some security features so if it discovers a file name that doesn’t end in m4a or if it encounters a folder name, then it could just bypass and go on to the next record.  Lots of things that can be done and maybe I might get round to them, or you can.

This project helped me learn a lot of things.  It gave me confidence in writing Perl scripts.  It helped me learn a lot about the interaction between audio/video files and the apps that store and play them.  It also helped me learn more about using Unix/Linux and the Android OS.  Even though for me this is a one-shot app, it has been good for learning more about computers and the languages that program them.  Hack on!

Establish files and folders for your Certification Authority – OpenSSL

Since the actual number of instructions in this post will be fairly short, I am going to spend some time on the why’s and wherefore’s of setting up your own self-signing certification authority.  If you don’t care about the philosophy part, you can skip on down the page to the instructions.

My base philosophy on anything to do with personal computing is in the word personal.  When I began working with a personal computer, the general concept most of us had was that the computer was to be a tool that we configured to do our bidding.  The person that operated a personal computer would determine what he or she wanted the computer to do then program it to do so.  Businesses bought large computers (whether mini or mainframe) and those computers were bought with their software (and sometimes with technicians on-site) to do things for businesses.  But the personal computer was supposed to be personal.

That all changed when Bill Gates and Microsoft came along.  It was through his influence on the PC market that the idea that a person just bought ready-made software to do what they wanted to accomplish.  Somehow, that has also transferred over the decades to all forms of computing, even to servers.  With the arrival of security and encryption needs, the idea has sprung up that only a third-party signer can do the job.  And, under some circumstances, that is correct.

If you are going to run an e-commerce site where financial transactions are taking place, you need the security and peace of mind that your site is confirmed by a third-party signer.  Where money is involved, don’t take a shortcut.  However, if your primary purpose is the encryption provided by that certificate, paying money to a third-part signer is probably not all that worthwhile.  This is my case.  I don’t really need the proof that my site is “approved” by a third party.  What I needed for e-mail is the encryption provided by the SSL layer so that our logons couldn’t be easily discovered.  If our only access to e-mail was going to be internal to our network, I would not have needed even that much.  However, with staff that use webmail, remote clients, IPhones and Android phones, I could never tell who might be out there.

I do take offense that those on the help forums that simply tell folks to put up their money and go get a third-party signed certificate.  First, it’s wrong to dictate to people that the only solution is the one that costs money.  Second, to establish a defacto standard is one thing, to require everyone to obey it is a different story.  As an example, I clicked on a story today about a Marine pilot on a page that belongs to the U.S. military.  It was a DoD publicity site used by the Marines to publish informational stories for the public.  However, my Android browser alerted because the certificate wasn’t trusted.  Why wasn’t it trusted?  Because it was a self-signed certificate.  Who was the signer with all the right information in the certificate?  The Department of Defense.

Let me explain how this works.  The reason that third-party signers are so set up with their market niche is because they have already gotten their root certificates accepted by the browsers of the world.  If your browser reads a certificate that it already has a root certificate (or link to one) then it can look down that root’s tree and find the server’s certificate.  The only reason that the DoD page’s certificate wasn’t automatically accepted was that the browser didn’t have the DoD’s root certificate.  But, the DoD is large enough and secure enough shouldn’t it’s root certificate already be there?  Just for grins, I also checked the root certificates on my own Linux desktop and the DoD doesn’t appear to have one listed.

Therefore, I feel no problem with showing anyone how to make their own certificate authority or how to use their own self-signed certificates for any uses that are not connected with e-commerce.

Now, on to the lesson.  After you have installed your OpenSSL software, the next thing to do is establish the folders and files you are going to need to create your CA, or Certificate Authority.  Even though you aren’t probably going to sign certificates for others (actually, I hope you don’t), you will need a place to keep track of your root and server certificates.

First create your own CA folder.  You can call it myCA or domainCA or whatever you like.  But, I would use the CA to remind you it’s your certificate authority.  OpenSSL will create an /etc/ssl folder and you could create it there, but you don’t have to.  I put mine in my home folder so that it’s close.

$/myhome/>mkdir myCA

Now, under this folder you will need a place to keep all your certificates after signing, a file for the indexes on certificates and a serial number file.  You should also make a root protected folder called private to store your  private keys.  Your public keys (in your certificates) have to be world readable.  You will also need a local tmp directory for signing purposes.

$/myhome/>cd myCA

$/myhome/>mkdir certs

$/myhome/>mkdir private

$/myhome/>mkdir tmp

$/myhome/>touch index.txt

$/myhome/>echo "00a">serialfile

Now, we should have a working OpenSSL installation and an accessible set of folders and beginning files to start our CA.  Next, we want to establish the configuration file for our root certificate.

How to Install OpenSSL on a Linux Server (debian)

OK, friends.  This post will be short.  Installing OpenSSL for most Linux heads will be easy.  There isn’t anything to do in most cases.  Why?  OpenSSL is provided in many Linux distributions because you need it for servers and desktops (that’s how you get to https:// sites).

To check and see if OpenSSL is already installed on your machine, type the following:

$>openssl version

If you get a response like the following,

$>OpenSSL 0.9.8o 01 Jun 2010

(or similar), then your openssl installation is just fine. You can go to the next step (when I write it).

However, if you are one of the few that don’t have OpenSSL installed, then use the following (from here on , I am assuming you are on a debian-style system.  Sorry ’bout that Redhat.):

$>sudo aptitude install openssl

There are also some libraries that may need installing.  If you are on Ubuntu, go check the following to get the whole scoop of installation

As soon as you are certain that OpenSSl is installed, we can proceed to the next step in the process, establishing the files and folders for your CA.

Ubuntu, SSL, and how to overcome Microsoft problems

It has been a really long time since my last post.  However, I have been busy.  Someday, I am going to have to get all these things I’ve worked on written down.

My most recent situation has urged me to carve out the time to post my work so that a) others can benefit, and b) I can remember what I did.

Part of my responsibilities for my work has been to oversee the e-mail system of our organization.  As numbers of servers and users go, it is small.  We have one e-mail server running on Ubuntu Linux.  The e-mail software is a combination of Postfix and Dovecot.  Postfix is the MTA (mail transport agent) and Dovecot is the MUA (mail user agent).  The MTA has the primary responsibility for getting and sending all the mail that we address to others and that others in the world address to us.  The MUA handles our users’ connectivity to get their mail into and out of the system.  We currently have no more than a dozen actual users with mailboxes on the system, so for most purposes, it is not difficult to maintain.

Our purpose in setting up our own mail server versus using an online system (such as the one our domain-ISP provides) is not problems with our users, but problems with those that receive mail from us.  We are an organization that has a few in-house folks that send e-mail to our members.  The members are not users of our mail system but will have clients and providers such as AOL, Yahoo, or Verizon.  For a long time when I first began working in this capacity, no problems could be examined.  If a member said they didn’t get an e-mail, we couldn’t track anything along the path, nor did we get any kind of error messages from our provider’s mail servers.  Setting up and operating our own mail server has solved many of those problems.  They were solved because in the course of building and using the server, we had to learn all about how e-mail traverses the Internet and the standards that have come to play because of spam and phishing practices of nefarious people.

We had our mail server operational for some time.  Yet, the security on the MUA and MTA were still sketchy.  We did have TLS enabled on Postfix, but only a dummy certificate.  We did not use any of the encryption/security features of Dovecot.  Therefore, most of our logins were in plain text and could be known by middlemen wishing to do a little work.  It was our desire to engage SSL encryption on both the MTA and the MUA.  The use of SSL (Secure Sockets Layer) on both Postfix and Dovecot would enhance our mail security by encrypting the logins and communications of our users as they get and send e-mail.

Next, it is also important to describe our e-mail network.  While the server is Linux, there is only one other Linux desktop (mine).  The majority of our desktops are Windows XP Pro systems, with some Macs, and most users also have either an IPhone or an Android smart phone.  Each of these uses different e-mail clients and methods of dealing with SSL encryption.  But more on that soon.

After much trial and error, I finally discovered that there are several steps that must be done in order and correctly to make SSL encryption work on an e-mail server.  Most of it revolves around two items; the SSL certificates and the SSL public/private key pair.  And, it is important to note as we go along that I said certificates and not the singular certificate.

This tutorial will only deal with establishing self-signing certificates and authorities.  While it is true that a third-party certificate provides some added security in the real world, it is only really required when performing on-line commerce.  In our case, we chose to use self-signed certificates because 1) we really only needed the encryption feature, 2) third-party certificates are expensive, and 3) there is no LAW that says we have to use third-party certificates.

The progression of steps are:

1. Install OpenSSL and properly set it up.
2. Establish the files and folders for your CA (Certificate authority)
3. Create the root configuration file with the proper data.
4. Create the root certificate and private key.
5. Create the server’s certificate signing request and private key.
6. Sign the certificate.
7. Set up the appropriate ports in your firewall.
8. Connect from the MUA and deal with the self-signed certificate.
9. If you use Microsoft Outlook clients, install the root certificate in certificate stores before changing to SSL encryption

Each of the next several posts will examine and give detailed information on each step as we cover it.  However, it is also important to remember that I did not come up with this information all on my own.  The following sites or blogs contributed greatly to helping me understand and utilize SSL encryption and certificates.  As we go forward, please check them for the final say on what I am telling you.  I might get something wrong.  I will also welcome your comments and suggestions on how this goes.  Follow along.  We must might learn something.

SSL Shopper – SSL Checker

Debian Administration – Setting up an SSL site with Apache2

langui.sh – Creating a Self-Signed CA and server certificate

The OpenSSL Configuration File

Ubuntu Community Documentation – OpenSSL

Next step, getting OpenSSL installed.

Ubuntu Natty Narwhal Upgrade Review

Well, by now, most of us that use Ubuntu Linux desktop distribution have had to cross the Rubicon of upgrading to Natty Narwhal (11.04).  Mine happened Friday and I must say that I am not really impressed.

It seems that someone in the Ubuntu crowd decided to get their inner “Mac” on.  It’s either that, or Jobs and company have infiltrated Ubuntu in order to take over.  I’ve never been a Mac person but I have tried to work with a colleague on integrating their Mac into our Windows network.  Let’s just say it wasn’t pretty.

Why do I say this?  Because the look of the Unity desktop (and it’s associated quirks) is very much “Apple-ish”.  Instead of the cover slide which lurks at the bottom of most Macs, Unity’s runs up the side of the screen.  While it is a versatile way to work, Ubuntu got rid of the gnome-applet icons we could put all over our taskbars.  You know, those useful things like bug eyes and the weather.  Yes, I liked my weather, damnit!

The sliding cover show does work pretty well in finding your icons.  I thought something was broken at first because I couldn’t seem to move my icons around as had been foretold.  I figured it out today.  You have to click and hold on the icon you want to move before it will disconnect from the slider.  Then you can move it up or down to the location of your choice.  I did discover that a couple of them are fixed, like the application icon.

Another new feature of Unity that took getting used to is the act of keeping the program menu on the taskbar instead of locked onto the application.  This was made worse in my case because my theme left me with black letters on dark brown background.  My menus were invisible until I caught sight of the change during a mouse over.  In addition, you can’t see the menu at all unless you hover over the application name in the taskbar.  I had thought originally that Firefox had gone all Chrome on us.

It took me three days to finally figure out that Banshee was not going to work with my iPod.  While that is not Banshee’s fault (I still prefer to blame Jobs and company), it is lamentable.  I had such high hopes for Banshee.  Oh well.  Maybe someday, someone will overcome the Apple monster so I can turn off Windows.

While I realize most of my problem with Narwhal are cosmetic issues, please remember that this is a “Desktop” distribution of Linux.  Cosmetics count.  It is my staunch belief regarding desktop software and OS’s is this: If you have to read a book, look up in a manual, or use Google to figure out how to make it work, someone didn’t do their job correctly.

For the geeks, there are some differences under the hood too.  However, I haven’t seen any evidence of changes or problems in this area as of yet.  Ubuntu Natty Narwhal still seems to be as stable as any other distribution.

In the end, whether you choose to upgrade (or keep your upgrade) is a personal choice.  Despite my misgivings about the desktop, one eventually has to upgrade.  Whether you are a Mac, a PC, or  Linux, upgrades come around and our problems with things like cosmetics can be gotten around one way or another.  There are improvements that come to us in upgrades that are important to have, and sometimes are essential.  The folks at Canonical will get lots of feedback and the forums will be alive with comments, but eventually all this discussion will result in future changes.  Just hang on, it’s a bumpy ride.

Ubuntu Server and OpenLDAP…

After two days of hard work, I have finally gotten an LDAP server running (albeit only at square one) on Ubuntu Maverick server (I think it’s 10.04 LTS).  It has been a nightmare of going from one how-to to the next, spending time on forums, searching out set up needs and configuration needs; yet, nothing was working.  Each time that I tried to get the initial directory tree set up, I would get errors.  Most of which would occur after I get a backend loaded and it wouldn’t acknowledge the password.  But, now it is working.

Before I delve into this subject much further, let me share some background and one condition: this will not be a definitive how-to (I’m going to send you to where it is, though).  I handle the IT for my church.  I have two things that are quite important to watch over; e-mail and our e-mail updates.  I currently have our domain name registered through 1and1.com.  Though they have been decent, I have run into difficulties I can’t get answered.  In many IT systems, if a recipient outside your organization could not send or receive e-mail to you, it would be, “OK, sounds like you have a problem.”  When those folks are also your members, you can’t quit there.  In addition, we use a vendor in order to send out bulk e-mail (others call this spam, but some folks have a use for it and we are one of those).  Our provider has us “build” the e-mail on an on-line editor.  The person that does this task each day is growing increasingly frustrated because the vendor’s editor keeps changing our choices after saving the e-mail body.

To solve this, my plan is setting up an e-mail server using a relay host.  In addition, I am also putting together the software needed to host our contact list on the same server so that we can maintain our list in only one place instead of everyone’s PC.  For this, I am using Ubuntu (don’t bitch, I like it) Maverick, running Postfix for the MTA (Mail Transport Agent), and Dovecot to support the MUA’s (Mail User Agents or mail clients).  These I have pretty much done.  My DNS settings were slightly problematic in that I couldn’t get the DNS to respond to the correct server names so that the connections looked normal.  That took a little time and never got completely right, but it’s decent and it’s stable.  My next step was to set up an in-house LDAP server (Lightweight Directory Access Protocol) so that we could host one common address book (primarily because most of the mail clients are Outlook).  That’s where my two days of work comes in.

Let me say, that I will not bore you with everything that went on in those two days.  I’ll get right to the point of what it takes to make LDAP work on Ubuntu.  First, you don’t actually get an LDAP server.  Instead, you get and install it’s daemon: slapd, and associated utilities (ldap-utils).  DO NOT get them from the Ubuntu package repositories.  They do not get set up in order to follow the primary how-to that I am going to show you.  You will need to go to the source: OpenLDAP.

But before you do that, you need to get the following installed and running: Berkely DB engine and headers and gcc (if you haven’t done so already).  You can install the Berkeley DB package from the Ubuntu repositories, but make sure that you install the -dev version.  It also needs to be at least version 4.2.  It seems according my sources that you aren’t going to use the Berkely engine as you are the header files.  Only when the ‘-dev’ version is loaded does it also load the header files.  After these are installed, you can go to the OpenLDAP home page and click the link for their “Quick Start Guide”.

Carefully read this guide and follow through on each step.  This will not be a package installation.  Instead, you are going to download the TGZ file and install it manually.  Don’t worry though, it has a good set of instructions and self-configuration files to walk you through the process.  When you get to the ‘make’ and ‘install’ steps, be prepared to wait a while.  They take a long time to complete.

After you get slapd installed, OpenLDAP’s guide also walks you through getting it running (minimally) and getting your first directory tree built.  After that, you will have to set up your own init script in order to have the daemon started and stopped on boot.  I had quite a bit of difficulty in doing this, but finally succeeded through brute force.  If you know how to set up upstart jobs, this would be a better method.  There is a file called ‘skeleton’ in /etc/init.d that has the bones of a correct init script you can use.  I think you should add /usr/local/ or /usr/* to the PATH variable and it might work better.  Remember to run “sudo update-rc.d slapd defaults 90″ after you check out your init script.  That should allow Ubuntu to start and stop your daemon on boot (assuming you name your init script slapd).

I am not certain why this method works when other don’t.  I believe that some of the problem is there are many competing install possibilities and though some of the how-to’s got it to work, it’s not universal.  I am also of the belief that most of the package installs are defaulting to SSL and SASL as being “ON” in slapd, when those aren’t included out of the box for Ubuntu server.  For an “open-to-the-world” LDAP, I recommend that method.  However, you also need to be able to simplify initial installation so that you aren’t fighting multiple problems when just starting out.  In either case, on my Ubuntu Maverick server, following the directions found at OpenLDAP did work.  Now, all I have to do is learn the rest to get to addressbook stage.

Creating DVD’s in Ubuntu

I have just spent a good part of a week learning how to rip, convert, edit, author, and burn DVD’s.  I had a project where I needed to split several individual video lessons from one DVD and create individual DVD’s for each lesson.  I thought this might be a simple task, but I was owned.  Nothing involving DVD’s and video files appears simple at all.  I finally did succeed at gathering and collecting all I needed in order to make the DVD’s.  Partly for myself, and partly to share, I am making this post so that we have one place to look for the entire process.

Before I begin, I need to share the credit with the sites and people where I found resources (click on the links to go there):

genisoimage and growisofs – (These are at the bottom of the page)

dvdauthor

ffmpeg wiki (conversion to mpeg2 at bottom of page under DVD)

PiTiVi

If you look at the list of resources above, you will quickly see that we are  not going to be using a one step operation.  On the contrary, this will require several steps, some of which are optional.  If you are working in Ubuntu, most of the software is already installed.  The ones that are likely not installed are ffmpeg and dvdauthor, but your package manager can get them installed quickly.  The steps by summary are:

1. Load and locate DVD files (regular linux commands).

2. Convert VOB files to mpeg-2 files (ffmpeg).

3. Perform any editing needed and render to mpeg-2 files (PiTiVi).

4. Create folder (directory) to hold DVD structure and use dvdauthor to create that structure (dvdauthor).

5. Use genisoimage to create DVD compatible ISO files.

6. Use growisofs to burn ISO file to DVD.

Now, let’s take these in more detail:

Locate and load files from DVD –

While you can use a DVD ripper, such as AcidRip, some DVD’s will not rip well.  In particular, the ones I tried to rip could only identify the introduction video on a DVD of  8 individual lessons.  I would guess, that if you are ripping a movie DVD, AcidRip might be the better choice.  But, failing that, you can always do it this way.

Every drive mounts as a folder.  My DVD drives usually mount at:

/media/DVDVolume

Though sometimes, instead of DVDVolume, it will use the name or title of the DVD itself.  In either case, you can change to the directory and you’ll see something like this:

cd /media/DVDVolume
ls
AUDIO_TS     VIDEO_TS

Next, change to the VIDEO_TS folder where you would see something like this:

cd VIDEO_TS
ls
VIDEO_TS.BUP  VIDEO_TS.IFO  VTS_01_0.BUP  VTS_01_0.IFO  VTS_01_1.VOB

I chose the structure from a simple DVD I made.  Your real DVD structure, especially if it is a movie, could be lots more complicated.  However, the only files we are really concerned with are the VOB files.  I don’t know exactly what kind of file these are, but what I have seen so far is that they are just mpeg2 files that have the ending of VOB.  However, ffmpeg knows what to do with them and all we have to know is that we need to convert them to what we wish to use in a video editor, such as PiTiVi.

Originally, I converted them to AVI files, but I have recently discovered that you can run ffmpeg and convert them  straight to mpeg2 video files.  If you are the risky type, you might see if the VOB’s will drop directly into PiTiVi (I haven’t tried that yet).  At this point, I recommend making a folder to house your conversions, editing files, authoring folders, and your output ISO files.  I made a folder in my home drive called dvd; short and easy to type.

You should now be in the VIDEO_TS folder. All we need do is convert the individual files.  You would do the following:

ffmpeg -i XXX.VOB -hq -target dvd ~/dvd/XXX.mpeg

This should make sure that you have a good quality video file to work with in PiTiVi.  Please note that I have had trouble sometimes with the “-hq” option.  I haven’t seen it in the man pages for ffmpeg, but have seen it used.  If you have problems with the conversion, try taking it out.  Another legal type of target is also “ntsc-dvd” (if you are in the US, pal-dvd for others). Also, this is not meant to be a complete run down on ffmpeg.  There are too many to discuss at this time.  Finally, replace the /dvd/ and XXX with your folder and file names.  It is a good idea at this point to keep something close to the input name in the output file because you might need to know which order the VOB’s came off the DVD drive.

Edit and render videos as needed–

At this point, you can drop your videos into PiTiVi or the video editor of your choice. Because this is not always something you need to accomplish and the discussion of video editing is larger than the space I have here, we won’t be discussing video editing or its associated software. Except for one point. When you are ready to render, set your output file to be rendered as an mpeg2, if at all possible. This will put you as close to the final type of file we will use next and will save extra time and effort lost in doing conversions.

Author your DVD structure–

Now, that you have your video ready to burn, we have two steps to do before we get there. First, make sure that you have created an empty folder in the same folder as your video file. This will become the folder that dvdauthor uses to build the dvd structure. I tended to make ones called “disk1″ or some such. Second, you need to make an XML file in the same folder as your video. You can use your editor of choice, but it should contain what you find at this link.  It seems that there is not a way to represent XML notation on WordPress.com, so I am referring you to the sourceforge page.

Save this with a filename that ends with XML and make sure it is in the same folder as your video file. Replace the “myfilename.mpg” with the file name of your video. Again, there is much more that dvdauthor can do, but for that, you will need to see the link above to the dvdauthor website. WARNING: This XML file will only work for a DVD with a single video file. There are different settings for the XML file if you use multiple video files.

When you have your video file, xml file, and empty directory ready, you are ready to create the DVD structure with dvdauthor.  You do that with the following command:

dvdauthor -o disk1 -x disk1.xml

Remember to replace “disk1″ with the name of your empty folder and the name of your XML file.  When this successfully completes, your folder will contain an AUDIO_TS and VIDEO_TS folder.  VIDEO_TS will have a structure similar to the one printed out above with just a single VOB file.  You are now ready to make the ISO image for burning.

Make the ISO image for burning–

Because of the structure of CD and DVD drives, one cannot simply copy files over to them.  Yes, you can set up some of the RW type of drives for something similar, but if you are making an actual audio (CD) or video (DVD) disk, you much create an ISO image file.  To do that, use genisoimage with the following:

genisoimage -R -J -dvd-video -o disk1.iso disk1

Again, change the “disk1″ to the names you are using.  A little bit of explanation here is important.  genisoimage functions a little like ffmpeg in that you have to tell it what kind of DVD  you are making.  The “-R” and “-J” are used for long filename recognition.  The “-dvd-video” tells it to make an actual, playable-in-a-standalone-DVD disc.  The “-o” option tells us what filename for the output and the “disk1″ at the end is the name of the folder where our DVD structure is located.

Burn the silly thing–

You are almost there.  One more step is left, and it is the most crucial; getting this burned on a disk.  Insert your blank DVD in your writer and when it is properly recognized, go for the following:

growisofs -speed=2 -dvd-compat -Z /dev/scd1=disk1.iso

This will have a lot of output telling you where you are in the process.  You will not be finished until the disk pops out of the drive.  The “-speed=2″ is optional, but recommended by others.  The “-dvd-compat” makes the actual DVD-ready disk.  The “-Z” I am unsure of , but it’s necessary.  The “/dev/scd1″ might fool you, and yours might not be the same.  To find out what the name of your DVD-writer is, use:

wodim --devices

This will give you a list of CD/DVD devices on your system and most have only one.  However, if you choose wrong, you’ll know it immediately after hitting the “Enter” key and you can change it on the next try.  Replace “disk1.iso” with the name of the ISO file you made with genisoimage.

If your disk pops out, you should be finished.  Push it back into the drive and your movie player should automatically take off and begin to play the DVD.  Plug it into a DVD player to test it there. If it works in both places, you did everything correctly, and you’re now a certified, freakin’ genius.

What to do if things go wrong?  First, most of these routines will output lots of information.  Try and watch some of what they are doing and you’ll likely see the cause.  Be sure everything gets typed in accurately.  If at all possible, work from a clean folder with just the files you need.  It clears up clutter.  After every step where you run a conversion, you can run mplayer to see if the video is working.  If it will play through mplayer, the file should be fine.

What to do next?  If you’re industrious, you could find a way to script this so that you can do things in one step.  I intend to as time arranges (laughs).  If you’re really industrious, you might create a Gtk or python wrapper for it and make it into a real windows-type application.  In any case, enjoy using these instructions and I make no claim that they’ll work on everyone’s machine.  You are also on your own as far as legality goes.  Make sure you have the legal right to rip a DVD before you do so.

This took a good bit of time to research, but it was well worth the effort.  I learned a lot about video processing and my hardware while running on Ubuntu linux.  Creating DVD’s is not a simple task.  In open source, I am not sure there is a single application that does everything at one time.  In addition, I have learned that there are many different ways to create and burn DVD’s.  I chose to display these steps because I know they worked for me.  Dependent upon hardware types and what you wish to do, you might need different options for each application.  But, it is not impossible and that’s the important point.