VMware Backups with Veeam
I’ve recently deployed Veeam Backup to backup our 500 VMs, which all sit on a VMware vSphere.
We needed to monitor the backups to check everything is going okay. However, due to Veeam’s excessive keenness in reporting warnings where they aren’t needed, it’s useless trying to wade through all the pretty Veeam backup reports every morning.
What we needed was a way of telling the age of the most recent successful backup of each VM, which we could then pass on to some other tool for parsing (Nagios in our case, though the data’s the important bit!).
So I created a script which looks through every VM Veeam has ever backed up (i.e. every VM for which it has ever attempted a backup) and reports the age of the most recent backup of that VM. I wrote it in PowerShell, and you are welcome to download it and use it yourself. It generates a CSV table which you can read into anything. In my case it squirts it across the network to a waiting Nagios server, but you can do anything you like with it.
To send the CSV text it generates to the remote server it uses the Send-TcpRequest.ps1 script that I heavily modified from the Windows PowerShell Cookbook published by O’Reilly. I hope they don’t mind me messing with their code a lot!
We needed to monitor the backups to check everything is going okay. However, due to Veeam’s excessive keenness in reporting warnings where they aren’t needed, it’s useless trying to wade through all the pretty Veeam backup reports every morning.
What we needed was a way of telling the age of the most recent successful backup of each VM, which we could then pass on to some other tool for parsing (Nagios in our case, though the data’s the important bit!).
So I created a script which looks through every VM Veeam has ever backed up (i.e. every VM for which it has ever attempted a backup) and reports the age of the most recent backup of that VM. I wrote it in PowerShell, and you are welcome to download it and use it yourself. It generates a CSV table which you can read into anything. In my case it squirts it across the network to a waiting Nagios server, but you can do anything you like with it.
To send the CSV text it generates to the remote server it uses the Send-TcpRequest.ps1 script that I heavily modified from the Windows PowerShell Cookbook published by O’Reilly. I hope they don’t mind me messing with their code a lot!
Comments
Re-routing Traffic via a Remote Server
15/02/12 16:18 Filed in: Linux
This isn't a great long guide to using ssh for port forwarding, it just covers the one instance I needed.
Normally I connect my mail program running on c1 (my Mac at home) to server s3 port p3 (my mail server at work).
But today I can't get traffic from c1 to s3 because s3's corporate firewall has blocked me. So I need to re-route all the traffic destined for s3 port p3 via a separate server s2, which fortunately is in another country and I have ssh access to it.
You change your local mail program, instead of connecting to s3 port p3, to connect to "localhost" port p1 (some port on your Mac/PC you haven't used for anything else, e.g. 2000).
Then in a Terminal window you do
ssh -v -L p1:s3:p3@s2
To give you a concrete example, my IMAP server I want to talk to is at imap.work.com port 993.
My remote server (to which I have ssh access) which I want to route all traffic via, is called ssh.mydomain.com and it listens on port 22 as usual. I have an account there with username jkf.
On my local Mac running my Mail client, I've decided to route IMAP traffic via port 2000 as it's an unused port. That "2000" is pretty much any number in the range 1024-65535 that seems to work. If you hit one that doesn't work or produces an error, guess again.
I would type this:
ssh -v -L 2000:imap.work.com:993 jkf@ssh.mydomain.com
If the ssh server on ssh.mydomain.com ran on port 2222 instead of the default port 22, I would just have to add "-p 2222" to that command.
So your traffic from your mail program (set to talk to localhost port p1 which is 2000 in my case) does this:
c0 ==> s1 (==localhost) port p1 ==> s2 port 22 ==> s3 port p3
You will need to set up more than 1 of these as you'll probably want to re-route SMTP traffic as well. So just open another Terminal window and run another similar command in there, for the appropriate SMTP port.
The commands will look like they have just logged you in to your remote ssh server (ssh.mydomain.com) but they will have set up all the magic before they did it. The "-v" in the ssh commands makes it show you more detail of what is going on.
Normally I connect my mail program running on c1 (my Mac at home) to server s3 port p3 (my mail server at work).
But today I can't get traffic from c1 to s3 because s3's corporate firewall has blocked me. So I need to re-route all the traffic destined for s3 port p3 via a separate server s2, which fortunately is in another country and I have ssh access to it.
You change your local mail program, instead of connecting to s3 port p3, to connect to "localhost" port p1 (some port on your Mac/PC you haven't used for anything else, e.g. 2000).
Then in a Terminal window you do
ssh -v -L p1:s3:p3
To give you a concrete example, my IMAP server I want to talk to is at imap.work.com port 993.
My remote server (to which I have ssh access) which I want to route all traffic via, is called ssh.mydomain.com and it listens on port 22 as usual. I have an account there with username jkf.
On my local Mac running my Mail client, I've decided to route IMAP traffic via port 2000 as it's an unused port. That "2000" is pretty much any number in the range 1024-65535 that seems to work. If you hit one that doesn't work or produces an error, guess again.
I would type this:
ssh -v -L 2000:imap.work.com:993 jkf@ssh.mydomain.com
If the ssh server on ssh.mydomain.com ran on port 2222 instead of the default port 22, I would just have to add "-p 2222" to that command.
So your traffic from your mail program (set to talk to localhost port p1 which is 2000 in my case) does this:
c0 ==> s1 (==localhost) port p1 ==> s2 port 22 ==> s3 port p3
You will need to set up more than 1 of these as you'll probably want to re-route SMTP traffic as well. So just open another Terminal window and run another similar command in there, for the appropriate SMTP port.
The commands will look like they have just logged you in to your remote ssh server (ssh.mydomain.com) but they will have set up all the magic before they did it. The "-v" in the ssh commands makes it show you more detail of what is going on.
Force Linux to Scan for SCSI disks
07/02/12 21:51 Filed in: Linux
Many modern kernels will automatically notice new hard disks being added, and a simple
fdisk -l
will reveal them.
However, if that doesn't list them, do this:
echo "- - -" > /sys/class/scsi_host/host0/scan
echo "- - -" > /sys/class/scsi_host/host1/scan
echo "- - -" > /sys/class/scsi_host/host2/scan
fdisk -l
and you should now see them.
fdisk -l
will reveal them.
However, if that doesn't list them, do this:
echo "- - -" > /sys/class/scsi_host/host0/scan
echo "- - -" > /sys/class/scsi_host/host1/scan
echo "- - -" > /sys/class/scsi_host/host2/scan
fdisk -l
and you should now see them.
Creating Lists in Sympa
Creating a list in Sympa involves a whole host of files interacting with each other, and all sorts of configuration snippets are used to create the filename of a resource needed to build the list. This attempts to describe what is taken from where, when a list is created from the command-line with a “sympa.pl --create_list” command.
The command (run as root) is
sympa.pl --create_list --input_file=<xml-file>
<xml-file> is a copy of /etc/sympa/sotontemplate.xml with tokens substituted in to set values which are inserted into the template config file. The <xml-file> can be deleted immediately after the list is created, it is not needed.
The <xml-file> looks like this (note the *1* and so on for footnotes):
Notes:
The XML file is meshed into the template config file in /etc/sympa/create_list_templates/<type>/config.tt2 and then interpreted as a TT2 template (see www.tt2.org), and the output goes into the list's own individual config file which is at
/var/lib/sympa/list_data/all.soton.ac.uk/<listname>/config
That file is compiled by Sympa automatically when needed. It can be edited manually at any time, and Sympa will automatically recompile it as and when needed, without you needing to tell it about the change.
Note that the syntax of the template config file config.tt2 is very strict. It is constructed of a series of "paragraphs" which are a bunch of lines, with a blank line terminating the paragraph. It does not support comments, blank lines are *very* important, and leading white space on a line breaks everything. You have been warned!
"Editors" is what Sympa calls list moderators. When they are used depends on the setting of the "send" parameter in the list config file. See the section on "Authorisation Scenarios".
The editors of a list are not statically defined, but are pulled from a SQL database query as needed. The data is cached by Sympa for speed.
If the list has any editors, the editors are defined in the list's config file by a "paragraph" like this:
Notes:
The data source file looks like this:
Note this file *does* handle leading white space on lines.
Most of this is obvious, and it can handle any <db_type> that is available in the Perl DBI modules.
The "source_parameters" from the list's config file are received as the "param" array, and the data source file is again a TT2 file (see www.tt2.org).
The default editor address for every list is “postmaster@ecs.soton.ac.uk".
"Subscribers" are what Sympa calls list members. Only list subscribers receive a copy of postings to the list. So people can be able to post to a list without being able to receive a copy of their own posting. Posters who are subscribers always receive a copy of their own posting.
The subscriber list for a list is generated by an SQL query defined directly in the list's config file, it does not use a .incl file like the editors. The paragraphs defining the subscribers (taken from the template config file, not a list's real config file) look like this:
Notes:
The list of subscribers for each list is cached by Sympa for a length of time defined by the /etc/sympa/sympa.conf setting "default_ttl" and is set to 3600 seconds (1 hour) at the time of writing. So all changes to the table of list subscribers in the "all_lists" database will take up to 1 hour to propagate to the lists. There is no way for force an update of a list's subscribers.
For the purposes of my project, virtually everything is closed and blocked, as users don't need to be able to use the capabilities of the email-driven or web-driven interface to manage lists, as they are all done automatically anyway. The only time authorisation scenari are used is to control who can post to a list.
The people that can post to a list are defined in the list's config file by the paragraph:
/etc/sympa/scenari/send.<scenario-name>
There is also a zero-length file
/etc/sympa/scenari/send.<scenario-name>:ignore
which tells Sympa not to tell anyone about the name or details of the scenario in the web management interface (which we're not using anyway).
The scenario file looks like this:
No comments are allowed. The syntax is strict. Each rule contains a condition, a list of methods, "->" and an action. The "smtp" method is the one we're interested in, it takes the address from the "From:" line of the email message. Rule conditions are matched in order from top to bottom, and the first matching rule is used, at which point rule processing stops.
This returns true or false. If true and the method used is listed, the action happened. You can include single-quoted strings like 'uos-staff', aliases for the name of the list, address of the sender and so on, function calls, all sorts of stuff. The '[sender]' can also be a regexp like '/\@soton\.ac\.uk$/' to cause regexp matches instead of simple string comparison. You can also calls Perl functions, SQL queries, LDAP queries, all sorts of stuff. In a Perl condition, the name of the file is taken from "CustomCondition::<custom-filename>" and the file is located in
/etc/sympa/custom_conditions/<custom-filename>.pm
In this example the list of "Received:" headers is passed to it as a parameter. If it returns 1, the action is executed. If it returns 0, the action is not executed. If it returns undef, there was an error. Changes to the <custom-filename>.pm file only take effect if Sympa is restarted with "service sympa restart".
I have created a send.<scenario-name> file for each type of list we use. Sympa does not notice changes to the scenari unless the datestamp on the list's config file is updated, at which point it updates the scenari for that one list. If you change a scenario file, you have to update the datestamp of the config files for all the lists that use it, forcing on-the-fly recompilation of the lists' config files.
You can change every bit of text ever output by Sympa, and the whole thing is multilingual too. I have stuck with en_US.
All the text output by email to users is contained in the TT2 files in
/etc/sympa/mail_tt2
Changes to files in this directory are automatically noticed by Sympa and processed immediately. Note this does not apply to the file "sympa.po" below.
For example, the "<reason_code>" above, used as a parameter to the "reject()" action, is defined in
/etc/sympa/mail_tt2/authorization_reject.tt2
The best way of finding which file contains the string you are trying to change is to search for a small bit of it in all the mail_tt2/*.tt2 files. Beware the actual text output will not be in these files, but something very close to it will be if you are working in English. Continue reading to find out why...
Let's take the example of the <reason_code> being "send_building". The "reject()" output messages are all defined in the file mentioned above, which is basically a huge "if...then...elsif...elsif...else" statement with hundreds of conditional expressions defined to test the "reason" parameter it is passed by Sympa.
Look through that file and find the "send_building" "ELSIF" condition. That will produce this bit of code:
The actual text output to the user is taken by internationalising the string "send_building", which is passed to the "loc()" function which does the translation. The "loc()" function can have parameters passed to it which are used to insert various bits of text into the translated text. Most of the original text to be translated is *almost* exactly the same as the output of the English translation, but not quite. More fun that way round. A few greps will find the text you're looking to change.
The translation (into better English) is done using the file
/usr/share/locale/en_US/LC_MESSAGES/sympa.mo
This file is a compiled version of the file
/etc/sympa/sympa.po
You can compile it by running
cd /etc/sympa && msgfmt -c -o /usr/share/locale/en_US/LC_MESSAGES/sympa.mo sympa.po
In sympa.po there are multiple definitions which look like this:
service sympa restart
Changes will not take effect until Sympa restarts.
Many mailing lists email all their subscribers automatically at the start of each month telling them they are subscribed to that list, and how to unsubscribe and post successfully. The frequency of such actions are given in the list's config file by the paragraphs:
/etc/sympa/list_task_models/<action>.<task_model>.task
so "expire_task never" is described in
/etc/sympa/list_task_models/expire.never.task
In my project expiry and reminders are not needed nor wanted. So to stop them working, their frequencies have been set to 1000 years. The reason for this is fairly obvious if you look at the structure and syntax of one of the task model files.
The command (run as root) is
sympa.pl --create_list --input_file=<xml-file>
<xml-file> is a copy of /etc/sympa/sotontemplate.xml with tokens substituted in to set values which are inserted into the template config file. The <xml-file> can be deleted immediately after the list is created, it is not needed.
The <xml-file> looks like this (note the *1* and so on for footnotes):
<?xml version="1.0" ?>
<list>
<listname>comp3020</listname> *1*
<type>soton</type> *2*
<listtitle>comp3020</listtitle> *3*
<subject>Students enrolled on COMP3020</subject>
<editor>false</editor>
<replypolicy>other_email</replypolicy>
<replyaddress>god@ecs.soton.ac.uk</replyaddress>
<tag>true</tag>
<ismodule>true</ismodule>
<isbuilding>false</isbuilding>
<isuos>false</isuos>
<owner multiple="1"><email>jkf@ecs.soton.ac.uk</email></owner> *4*
</list>
Notes:
- Sets the name of the list. Must be set in the XML file, not the template config file.
- Sets the value of "type" which is used to determine the location of the template config file used to construct the list's own config file. Must be set in the XML file. Location is /etc/sympa/create_list_templates/<type>/config.tt2
- These all set lower-case-named variables which are referenced in the template config file just mentioned.
- This sets the owner(s) of the list. Must be set in the XML file, not the template config file.
- Any characters such as “&”, "<" and ">" must be encoded as XML entities “&", “<” and “>” or it errors out.
The XML file is meshed into the template config file in /etc/sympa/create_list_templates/<type>/config.tt2 and then interpreted as a TT2 template (see www.tt2.org), and the output goes into the list's own individual config file which is at
/var/lib/sympa/list_data/all.soton.ac.uk/<listname>/config
That file is compiled by Sympa automatically when needed. It can be edited manually at any time, and Sympa will automatically recompile it as and when needed, without you needing to tell it about the change.
Note that the syntax of the template config file config.tt2 is very strict. It is constructed of a series of "paragraphs" which are a bunch of lines, with a blank line terminating the paragraph. It does not support comments, blank lines are *very* important, and leading white space on a line breaks everything. You have been warned!
Editors / Moderators
"Editors" is what Sympa calls list moderators. When they are used depends on the setting of the "send" parameter in the list config file. See the section on "Authorisation Scenarios".
The editors of a list are not statically defined, but are pulled from a SQL database query as needed. The data is cached by Sympa for speed.
If the list has any editors, the editors are defined in the list's config file by a "paragraph" like this:
editor_include
source editors *1*
reception mail
visibility conceal
source_parameters fpas-ecs-sys *2*
Notes:
- The <source> name is used to tell Sympa where to get the SQL query that will produce a list of the email addresses of the editors. The SQL query is defined in a data source file at
/etc/sympa/data_sources/<source>.incl - The comma-separated list of source_parameters are passed into the SQL data source file as param.0, param.1 and so on.
The data source file looks like this:
include_sql_query
db_type mysql
host localhost
db_name all_lists
user secretusername!
passwd secretpassword!
sql_query SELECT email FROM list_moderators WHERE list_address='[% param.0 %]'
Note this file *does* handle leading white space on lines.
Most of this is obvious, and it can handle any <db_type> that is available in the Perl DBI modules.
The "source_parameters" from the list's config file are received as the "param" array, and the data source file is again a TT2 file (see www.tt2.org).
The default editor address for every list is “postmaster@ecs.soton.ac.uk".
Subscribers / Members
"Subscribers" are what Sympa calls list members. Only list subscribers receive a copy of postings to the list. So people can be able to post to a list without being able to receive a copy of their own posting. Posters who are subscribers always receive a copy of their own posting.
The subscriber list for a list is generated by an SQL query defined directly in the list's config file, it does not use a .incl file like the editors. The paragraphs defining the subscribers (taken from the template config file, not a list's real config file) look like this:
user_data_source include2 *1*
include_sql_queryname subscribers *2*
db_type mysql *3*
host localhost
db_name all_lists
user secretusername!
passwd secretpassword!
sql_query SELECT email FROM list_members WHERE list_address='[% listname %]' *4*
Notes:
- This tells Sympa what sort of data source it is. Only "include2" exists in Sympa now.
- Note this is for information only, and has no apparent effect.
- The database type can be anything supported by the Perl DBI modules.
- The [% listname %] is replaced with the name of the list when the list is created, it cannot be written indirectly like that in the list's own config file.
The list of subscribers for each list is cached by Sympa for a length of time defined by the /etc/sympa/sympa.conf setting "default_ttl" and is set to 3600 seconds (1 hour) at the time of writing. So all changes to the table of list subscribers in the "all_lists" database will take up to 1 hour to propagate to the lists. There is no way for force an update of a list's subscribers.
Authorisation Scenari (Who Can Post?)
For the purposes of my project, virtually everything is closed and blocked, as users don't need to be able to use the capabilities of the email-driven or web-driven interface to manage lists, as they are all done automatically anyway. The only time authorisation scenari are used is to control who can post to a list.
The people that can post to a list are defined in the list's config file by the paragraph:
send <scenario-name>
where <scenario-name> tells it to read the rules of the scenario from/etc/sympa/scenari/send.<scenario-name>
There is also a zero-length file
/etc/sympa/scenari/send.<scenario-name>:ignore
which tells Sympa not to tell anyone about the name or details of the scenario in the web management interface (which we're not using anyway).
The scenario file looks like this:
title.gettext Building list, Private, moderated for University staff
CustomCondition::not_from_campus([msg_header->Received]) smtp,dkim,smime,md5 -> reject(reason='from_offcampus')
is_subscriber([listname],[sender]) smtp,dkim,md5,smime -> do_it
is_editor([listname],[sender]) smtp,dkim,md5,smime -> do_it
is_subscriber('uos-staff',[sender]) smtp,dkim,md5,smime -> editorkey
true() smtp,dkim,smime,md5 -> reject(reason='send_building')
No comments are allowed. The syntax is strict. Each rule contains a condition, a list of methods, "->" and an action. The "smtp" method is the one we're interested in, it takes the address from the "From:" line of the email message. Rule conditions are matched in order from top to bottom, and the first matching rule is used, at which point rule processing stops.
Condition
This returns true or false. If true and the method used is listed, the action happened. You can include single-quoted strings like 'uos-staff', aliases for the name of the list, address of the sender and so on, function calls, all sorts of stuff. The '[sender]' can also be a regexp like '/\@soton\.ac\.uk$/' to cause regexp matches instead of simple string comparison. You can also calls Perl functions, SQL queries, LDAP queries, all sorts of stuff. In a Perl condition, the name of the file is taken from "CustomCondition::<custom-filename>" and the file is located in
/etc/sympa/custom_conditions/<custom-filename>.pm
In this example the list of "Received:" headers is passed to it as a parameter. If it returns 1, the action is executed. If it returns 0, the action is not executed. If it returns undef, there was an error. Changes to the <custom-filename>.pm file only take effect if Sympa is restarted with "service sympa restart".
Method
- "smtp" is the one we're interested in. This takes the [sender] from the "From:" line of the incoming email message.
Action
- "do_it" says the operation should be done (in this case, the message should be posted).
- "reject(reason='<reason_code>')" says the message should be rejected and a message sent to the sender explaining why. See the "Responses and Messages" section below for information about the <reason_code>.
- "editorkey" says the message should be saved locally and sent to all the editors of the list with an MD5 checksum. If the editor then sends Sympa the MD5 checksum in an email message telling it to proceed, the message will be posted. If the editor tells Sympa the message should be rejected, it is rejected (don't know what the sender is sent in this case). There are many other actions.
I have created a send.<scenario-name> file for each type of list we use. Sympa does not notice changes to the scenari unless the datestamp on the list's config file is updated, at which point it updates the scenari for that one list. If you change a scenario file, you have to update the datestamp of the config files for all the lists that use it, forcing on-the-fly recompilation of the lists' config files.
Responses and Messages
You can change every bit of text ever output by Sympa, and the whole thing is multilingual too. I have stuck with en_US.
All the text output by email to users is contained in the TT2 files in
/etc/sympa/mail_tt2
Changes to files in this directory are automatically noticed by Sympa and processed immediately. Note this does not apply to the file "sympa.po" below.
For example, the "<reason_code>" above, used as a parameter to the "reject()" action, is defined in
/etc/sympa/mail_tt2/authorization_reject.tt2
The best way of finding which file contains the string you are trying to change is to search for a small bit of it in all the mail_tt2/*.tt2 files. Beware the actual text output will not be in these files, but something very close to it will be if you are working in English. Continue reading to find out why...
Let's take the example of the <reason_code> being "send_building". The "reject()" output messages are all defined in the file mentioned above, which is basically a huge "if...then...elsif...elsif...else" statement with hundreds of conditional expressions defined to test the "reason" parameter it is passed by Sympa.
Look through that file and find the "send_building" "ELSIF" condition. That will produce this bit of code:
[% ELSIF reason == 'send_building' -%]
[%|loc()%]send_building[%END%]
The actual text output to the user is taken by internationalising the string "send_building", which is passed to the "loc()" function which does the translation. The "loc()" function can have parameters passed to it which are used to insert various bits of text into the translated text. Most of the original text to be translated is *almost* exactly the same as the output of the English translation, but not quite. More fun that way round. A few greps will find the text you're looking to change.
The translation (into better English) is done using the file
/usr/share/locale/en_US/LC_MESSAGES/sympa.mo
This file is a compiled version of the file
/etc/sympa/sympa.po
You can compile it by running
cd /etc/sympa && msgfmt -c -o /usr/share/locale/en_US/LC_MESSAGES/sympa.mo sympa.po
In sympa.po there are multiple definitions which look like this:
msgid "send_building"
msgstr "University Building lists are restricted to occupants of the building, and Estates and Facilities staff.\n\n"
"If you think you should be able to post to this list, use http://all.soton.ac.uk/search to search for yourself by staff/student number, and be sure you are sending from the correct address."
- msgid -- This is the text value passed to loc() in the file authorization_reject.tt2 described above.
- msgstr -- This line, and every line starting '"' immediately after it, define the actual text that is sent to the user.
service sympa restart
Changes will not take effect until Sympa restarts.
Regular Tasks
Many mailing lists email all their subscribers automatically at the start of each month telling them they are subscribed to that list, and how to unsubscribe and post successfully. The frequency of such actions are given in the list's config file by the paragraphs:
expire_task never
remind_task never
The frequency in the "<action>_task <task_model>" statement is described in the list task model file/etc/sympa/list_task_models/<action>.<task_model>.task
so "expire_task never" is described in
/etc/sympa/list_task_models/expire.never.task
In my project expiry and reminders are not needed nor wanted. So to stop them working, their frequencies have been set to 1000 years. The reason for this is fairly obvious if you look at the structure and syntax of one of the task model files.
VLC and Bluray discs
No-one much seems to know how to import Bluray discs using VLC (once you’ve gotten rid of the copy protection so you can actually make a backup!). So here’s how I have done it. This is all done with a recent copy of VLC on Windows. The user interface on the other versions of VLC is rather different, so your mileage will vary.
First of all, use AnyDVD HD to remove all the copy protection from the disc. Then use “AnyDVD Ripper” (comes with AnyDVD HD) to copy the useful bits of the disc onto your PC. You’ll need a fair bit of space, a Bluray disc can require several tens of gigabytes of space.
So now, in the “STREAM” directory within the ripped copy of the disc, you’ve got a bunch of *.m2ts files. The biggest of these will be the main feature film/movie. If AnyDVD HD did its stuff, VLC will be able to play the *.m2ts directly, and you need to open the m2ts in VLC to see the codec information. Particularly, check what video codec is used (H.264 makes life easier as then you won’t have to transcode the video), and note down the audio stream ID of the first audio track, or the first AC-3 audio track. It will often be a number like 4353. Also note the bitrate (e.g. 320kb/s) and the sample rate (e.g. 48000).
VLC will by default always use the first audio track, whether that will fit into the output format you have chosen or not. The result is you often get no audio in your output file at all. So go into Tools / Preferences, and show all settings (see the bottom left corner of the dialog). In the “Input / Codecs” first page, set the “Audio track ID” to the number you noted down earlier (my example was 4353). Save the preferences.
Go into the Tools menu and select “Messages” to view the messages window. Set the “Verbosity level” to 2 and press the “Clear” button. Leave this window on the screen, you will probably need it later.
In VLC, go to Media / “Convert/Save…”. Add the m2ts file. Choose “Convert” from the Open button. Set the output filename to something ending in “.mp4”. Edit the top settings profile “Video - H.264 + AAC (MP4)” by clicking the little spanners button, and note down all the settings, including any that are blank or 0. Return to the Convert dialog and click on the “New Profile” button (it’s to the right of the red “X” button). Enter everything you noted down, including all zeros and blanks.
In the “Video codec” page, tick “Keep original video track” if the codec of the original video was H.264. If it wasn’t, then tell it to convert it to H.264. Set the “Scale” to 1. You should copy the nitrate and frame rate from the original settings in the codec information you saw earlier.
In the “Audio codec” page, things get a little trickier. By default, VLC will always convert the first audio track it finds. If that is not “MPEG 4 Audio ( AAC )” then you will probably end up with no audio at all. By now you should have already set the audio track ID in VLC’s Preferences. So just choose “MPEG 4 Audio ( AAC )” here, and do not tick “Keep original audio track”. Set the nitrate and sample rate to the numbers you noted down earlier. Set the Channels to 2.
Save those settings and you will return to the “Convert” dialog box. Set the Destination file to wherever you want the output file to go, and make it end in “.mp4”.
Click “Start”.
It will probably race through the film, as it’s not having to transcode all the video. But when the progress bar gets to the end, it will sit there and hammer your hard disk. Let it continue, don’t press anything until the word “Streaming” has disappeared from the VLC window. Then it has finished.
Try playing your new video. If you are missing the audio, look back in the “Messages” window for any lines that mention “audio” or “a52” as that will tell you it is writing some audio. Also check the “Codec information” for your new video file and see if it wrote an audio track at all. If it didn’t, you might have the wrong audio Track ID set in the Preferences. You should get some clues as to why it didn’t work. Explaining how to fix all the problems that might arise is way beyond the scope of this document, sorry.
First of all, use AnyDVD HD to remove all the copy protection from the disc. Then use “AnyDVD Ripper” (comes with AnyDVD HD) to copy the useful bits of the disc onto your PC. You’ll need a fair bit of space, a Bluray disc can require several tens of gigabytes of space.
So now, in the “STREAM” directory within the ripped copy of the disc, you’ve got a bunch of *.m2ts files. The biggest of these will be the main feature film/movie. If AnyDVD HD did its stuff, VLC will be able to play the *.m2ts directly, and you need to open the m2ts in VLC to see the codec information. Particularly, check what video codec is used (H.264 makes life easier as then you won’t have to transcode the video), and note down the audio stream ID of the first audio track, or the first AC-3 audio track. It will often be a number like 4353. Also note the bitrate (e.g. 320kb/s) and the sample rate (e.g. 48000).
VLC will by default always use the first audio track, whether that will fit into the output format you have chosen or not. The result is you often get no audio in your output file at all. So go into Tools / Preferences, and show all settings (see the bottom left corner of the dialog). In the “Input / Codecs” first page, set the “Audio track ID” to the number you noted down earlier (my example was 4353). Save the preferences.
Go into the Tools menu and select “Messages” to view the messages window. Set the “Verbosity level” to 2 and press the “Clear” button. Leave this window on the screen, you will probably need it later.
In VLC, go to Media / “Convert/Save…”. Add the m2ts file. Choose “Convert” from the Open button. Set the output filename to something ending in “.mp4”. Edit the top settings profile “Video - H.264 + AAC (MP4)” by clicking the little spanners button, and note down all the settings, including any that are blank or 0. Return to the Convert dialog and click on the “New Profile” button (it’s to the right of the red “X” button). Enter everything you noted down, including all zeros and blanks.
In the “Video codec” page, tick “Keep original video track” if the codec of the original video was H.264. If it wasn’t, then tell it to convert it to H.264. Set the “Scale” to 1. You should copy the nitrate and frame rate from the original settings in the codec information you saw earlier.
In the “Audio codec” page, things get a little trickier. By default, VLC will always convert the first audio track it finds. If that is not “MPEG 4 Audio ( AAC )” then you will probably end up with no audio at all. By now you should have already set the audio track ID in VLC’s Preferences. So just choose “MPEG 4 Audio ( AAC )” here, and do not tick “Keep original audio track”. Set the nitrate and sample rate to the numbers you noted down earlier. Set the Channels to 2.
Save those settings and you will return to the “Convert” dialog box. Set the Destination file to wherever you want the output file to go, and make it end in “.mp4”.
Click “Start”.
It will probably race through the film, as it’s not having to transcode all the video. But when the progress bar gets to the end, it will sit there and hammer your hard disk. Let it continue, don’t press anything until the word “Streaming” has disappeared from the VLC window. Then it has finished.
Try playing your new video. If you are missing the audio, look back in the “Messages” window for any lines that mention “audio” or “a52” as that will tell you it is writing some audio. Also check the “Codec information” for your new video file and see if it wrote an audio track at all. If it didn’t, you might have the wrong audio Track ID set in the Preferences. You should get some clues as to why it didn’t work. Explaining how to fix all the problems that might arise is way beyond the scope of this document, sorry.